[jira] [Commented] (MAPREDUCE-5795) Job should be marked as Failed if it is recovered from commit.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941431#comment-13941431
]
Hadoop QA commented on MAPREDUCE-5795:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12635723/MAPREDUCE-5795.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4449//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4449//console
This message is automatically generated.
> Job should be marked as Failed if it is recovered from commit.
> --
>
> Key: MAPREDUCE-5795
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5795
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Xuan Gong
> Attachments: MAPREDUCE-5795.1.patch, MAPREDUCE-5795.2.patch
>
>
> If Resource manager is restarted when a job is in commit state, The job is
> not able to recovered after RM restart and it is marked as Killed.
> The job status should be Failed instead killed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5795) Job should be marked as Failed if it is recovered from commit.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated MAPREDUCE-5795: - Status: Patch Available (was: Open) > Job should be marked as Failed if it is recovered from commit. > -- > > Key: MAPREDUCE-5795 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5795 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Yesha Vora >Assignee: Xuan Gong > Attachments: MAPREDUCE-5795.1.patch, MAPREDUCE-5795.2.patch > > > If Resource manager is restarted when a job is in commit state, The job is > not able to recovered after RM restart and it is marked as Killed. > The job status should be Failed instead killed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5795) Job should be marked as Failed if it is recovered from commit.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941399#comment-13941399 ] Xuan Gong commented on MAPREDUCE-5795: -- bq .We cannot always mark the final state as FAILED too. It can be any of the final states. So, instead of passing errorHappenedShutDown to AMStartedEvent, you should pass forcedState. May be call the argument as forcedJobStateOnShutDown and then use it depend on whether it is set or pointing to null. DONE bq. Add tests to validate these other possible final states. DONE bq. There is no point in storing this extra information forcedJobStateOnShutDown into the history-file. So changes to Events.apr etc are not needed. We can localize this change only to the event and the ApplicationMaster. FIXED > Job should be marked as Failed if it is recovered from commit. > -- > > Key: MAPREDUCE-5795 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5795 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Yesha Vora >Assignee: Xuan Gong > Attachments: MAPREDUCE-5795.1.patch, MAPREDUCE-5795.2.patch > > > If Resource manager is restarted when a job is in commit state, The job is > not able to recovered after RM restart and it is marked as Killed. > The job status should be Failed instead killed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5795) Job should be marked as Failed if it is recovered from commit.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated MAPREDUCE-5795: - Attachment: MAPREDUCE-5795.2.patch > Job should be marked as Failed if it is recovered from commit. > -- > > Key: MAPREDUCE-5795 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5795 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Yesha Vora >Assignee: Xuan Gong > Attachments: MAPREDUCE-5795.1.patch, MAPREDUCE-5795.2.patch > > > If Resource manager is restarted when a job is in commit state, The job is > not able to recovered after RM restart and it is marked as Killed. > The job status should be Failed instead killed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-2349) speed up list[located]status calls from input formats
[ https://issues.apache.org/jira/browse/MAPREDUCE-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941343#comment-13941343 ] Hudson commented on MAPREDUCE-2349: --- SUCCESS: Integrated in Hadoop-trunk-Commit #5362 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5362/]) MAPREDUCE-2349. Modified FileInputFormat to be able to issue file and block location calls in parallel. Contributed by Siddharth Seth. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579515) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java > speed up list[located]status calls from input formats > - > > Key: MAPREDUCE-2349 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2349 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Reporter: Joydeep Sen Sarma >Assignee: Siddharth Seth > Fix For: 2.4.0 > > Attachments: MAPREDUCE-2349.1.wip.txt, MAPREDUCE-2349.2.txt, > MAPREDUCE-2349.3.txt, MAPREDUCE-2349.4.txt, MAPREDUCE-2349.5.txt > > > when a job has many input paths - listStatus - or the improved > listLocatedStatus - calls (invoked from the getSplits() method) can take a > long time. Most of the time is spent waiting for the previous call to > complete and then dispatching the next call. > This can be greatly speeded up by dispatching multiple calls at once (via > executors). If the same filesystem client is used - then the calls are much > better pipelined (since calls are serialized) and don't impose extra burden > on the namenode while at the same time greatly reducing the latency to the > client. In a simple test on non-peak hours, this resulted in the getSplits() > time reducing from about 3s to about 0.5s. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-2349) speed up list[located]status calls from input formats
[ https://issues.apache.org/jira/browse/MAPREDUCE-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-2349: --- Resolution: Fixed Fix Version/s: 2.4.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed this to trunk, branch-2 and branch-2.4. Thanks Sid! > speed up list[located]status calls from input formats > - > > Key: MAPREDUCE-2349 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2349 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Reporter: Joydeep Sen Sarma >Assignee: Siddharth Seth > Fix For: 2.4.0 > > Attachments: MAPREDUCE-2349.1.wip.txt, MAPREDUCE-2349.2.txt, > MAPREDUCE-2349.3.txt, MAPREDUCE-2349.4.txt, MAPREDUCE-2349.5.txt > > > when a job has many input paths - listStatus - or the improved > listLocatedStatus - calls (invoked from the getSplits() method) can take a > long time. Most of the time is spent waiting for the previous call to > complete and then dispatching the next call. > This can be greatly speeded up by dispatching multiple calls at once (via > executors). If the same filesystem client is used - then the calls are much > better pipelined (since calls are serialized) and don't impose extra burden > on the namenode while at the same time greatly reducing the latency to the > client. In a simple test on non-peak hours, this resulted in the getSplits() > time reducing from about 3s to about 0.5s. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-2349) speed up list[located]status calls from input formats
[
https://issues.apache.org/jira/browse/MAPREDUCE-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941329#comment-13941329
]
Hadoop QA commented on MAPREDUCE-2349:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12635675/MAPREDUCE-2349.5.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4448//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4448//console
This message is automatically generated.
> speed up list[located]status calls from input formats
> -
>
> Key: MAPREDUCE-2349
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2349
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
>Reporter: Joydeep Sen Sarma
>Assignee: Siddharth Seth
> Attachments: MAPREDUCE-2349.1.wip.txt, MAPREDUCE-2349.2.txt,
> MAPREDUCE-2349.3.txt, MAPREDUCE-2349.4.txt, MAPREDUCE-2349.5.txt
>
>
> when a job has many input paths - listStatus - or the improved
> listLocatedStatus - calls (invoked from the getSplits() method) can take a
> long time. Most of the time is spent waiting for the previous call to
> complete and then dispatching the next call.
> This can be greatly speeded up by dispatching multiple calls at once (via
> executors). If the same filesystem client is used - then the calls are much
> better pipelined (since calls are serialized) and don't impose extra burden
> on the namenode while at the same time greatly reducing the latency to the
> client. In a simple test on non-peak hours, this resulted in the getSplits()
> time reducing from about 3s to about 0.5s.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-2349) speed up list[located]status calls from input formats
[ https://issues.apache.org/jira/browse/MAPREDUCE-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941313#comment-13941313 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-2349: Okay, the latest patch looks good. Will check it in when Jenkins says okay.. > speed up list[located]status calls from input formats > - > > Key: MAPREDUCE-2349 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2349 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Reporter: Joydeep Sen Sarma >Assignee: Siddharth Seth > Attachments: MAPREDUCE-2349.1.wip.txt, MAPREDUCE-2349.2.txt, > MAPREDUCE-2349.3.txt, MAPREDUCE-2349.4.txt, MAPREDUCE-2349.5.txt > > > when a job has many input paths - listStatus - or the improved > listLocatedStatus - calls (invoked from the getSplits() method) can take a > long time. Most of the time is spent waiting for the previous call to > complete and then dispatching the next call. > This can be greatly speeded up by dispatching multiple calls at once (via > executors). If the same filesystem client is used - then the calls are much > better pipelined (since calls are serialized) and don't impose extra burden > on the namenode while at the same time greatly reducing the latency to the > client. In a simple test on non-peak hours, this resulted in the getSplits() > time reducing from about 3s to about 0.5s. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5603) Ability to disable FileInputFormat listLocatedStatus optimization to save client memory
[
https://issues.apache.org/jira/browse/MAPREDUCE-5603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941301#comment-13941301
]
Hadoop QA commented on MAPREDUCE-5603:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614446/MAPREDUCE-5603.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4446//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4446//console
This message is automatically generated.
> Ability to disable FileInputFormat listLocatedStatus optimization to save
> client memory
> ---
>
> Key: MAPREDUCE-5603
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5603
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client, mrv2
>Affects Versions: 0.23.10, 2.2.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Attachments: MAPREDUCE-5603.patch, MAPREDUCE-5603.patch
>
>
> It would be nice if users had the option to disable the listLocatedStatus
> optimization in FileInputFormat to save client memory.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[
https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941300#comment-13941300
]
Hadoop QA commented on MAPREDUCE-5787:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12635694/MAPREDUCE-5787-2.4.0-v5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4447//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4447//console
This message is automatically generated.
> Modify ShuffleHandler to support Keep-Alive
> ---
>
> Key: MAPREDUCE-5787
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: nodemanager
>Affects Versions: 2.4.0
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Critical
> Labels: ShuffleKeepalive
> Attachments: MAPREDUCE-5787-2.4.0-v2.patch,
> MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch,
> MAPREDUCE-5787-2.4.0-v5.patch, MAPREDUCE-5787-2.4.0.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Status: Patch Available (was: Open) > Modify ShuffleHandler to support Keep-Alive > --- > > Key: MAPREDUCE-5787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.4.0 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Critical > Labels: ShuffleKeepalive > Attachments: MAPREDUCE-5787-2.4.0-v2.patch, > MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch, > MAPREDUCE-5787-2.4.0-v5.patch, MAPREDUCE-5787-2.4.0.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5787) Modify ShuffleHandler to support Keep-Alive
[ https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated MAPREDUCE-5787: Attachment: MAPREDUCE-5787-2.4.0-v5.patch Incorporated review comments from Vinod >> Can we also change the MapReduce fetcher to use keep-alive depending on >> whether it is enabled or not? - HttpURLConnection will automatically use persistent connection when keep-alive and Content-Length headers are properly set. So, there is no need to change the fetcher code. >> Suggestion for Configuration renames - Fixed >>Add both to the mapred-default.xml - Fixed >>LOG KeepAliveParam along with other things like jobId, mapId etc. -Fixed >> populateHeaders. We are already parsing jobID, ApplicationId etc as part of >> sendMapOutput. We should avoid doing the string parsing multiple times. >> Is setting CONTENT_LENGTH important? Even so, for doing it, we are reading >> the index-record two times - Yes, content-length is very much needed for this. Fixed multiple parsing issue. >>Instead of re-defining new constants like CONNECTION_HEADER in >>ShuffleHandler, can you use the standard constants in java (HttpHeaders)? - Fixed >> Finally, can you reuse code between the two tests? - Fixed > Modify ShuffleHandler to support Keep-Alive > --- > > Key: MAPREDUCE-5787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.4.0 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Critical > Labels: ShuffleKeepalive > Attachments: MAPREDUCE-5787-2.4.0-v2.patch, > MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch, > MAPREDUCE-5787-2.4.0-v5.patch, MAPREDUCE-5787-2.4.0.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-2349) speed up list[located]status calls from input formats
[ https://issues.apache.org/jira/browse/MAPREDUCE-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-2349: -- Status: Patch Available (was: Open) > speed up list[located]status calls from input formats > - > > Key: MAPREDUCE-2349 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2349 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Reporter: Joydeep Sen Sarma >Assignee: Siddharth Seth > Attachments: MAPREDUCE-2349.1.wip.txt, MAPREDUCE-2349.2.txt, > MAPREDUCE-2349.3.txt, MAPREDUCE-2349.4.txt, MAPREDUCE-2349.5.txt > > > when a job has many input paths - listStatus - or the improved > listLocatedStatus - calls (invoked from the getSplits() method) can take a > long time. Most of the time is spent waiting for the previous call to > complete and then dispatching the next call. > This can be greatly speeded up by dispatching multiple calls at once (via > executors). If the same filesystem client is used - then the calls are much > better pipelined (since calls are serialized) and don't impose extra burden > on the namenode while at the same time greatly reducing the latency to the > client. In a simple test on non-peak hours, this resulted in the getSplits() > time reducing from about 3s to about 0.5s. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-2349) speed up list[located]status calls from input formats
[ https://issues.apache.org/jira/browse/MAPREDUCE-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-2349: -- Attachment: MAPREDUCE-2349.5.txt Thanks for the review. Updated patch attached. bq. Add the configs to mapred-default.xml as documentation? Done. bq. LIST_STATUS_NUM_THREADS_DEFAULT -> DEFAULT_LIST_STATUS_NUM_THREADS Done. bq. oldListStatus() -> singleThreadedListStatus() Done bq. Can you add a bit of javadoc to all the new classes and methods in LocatedFileStatusFetcher? Also to the main LocatedFileStatusFetcher class itself. Done bq. Synchronization needed for ProcessInitialInputPathResult.addError()? Not required. It's local to the specific instance. bq. Can you group the callable, result and call-back for each type of operation together in two classes? Moved the Result into the callable. The CallbackHandler is non static - so moving that requires a fair amount of change, and additional parameters. Have left that as is. bq. The 'result' variable doesn't need to be a class field of ProcessInputDirCallable. Similarly the one in ProcessInitialInputPathCallable. Made this local to the method. Also fixed a typo in one of the log messages > speed up list[located]status calls from input formats > - > > Key: MAPREDUCE-2349 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2349 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Reporter: Joydeep Sen Sarma >Assignee: Siddharth Seth > Attachments: MAPREDUCE-2349.1.wip.txt, MAPREDUCE-2349.2.txt, > MAPREDUCE-2349.3.txt, MAPREDUCE-2349.4.txt, MAPREDUCE-2349.5.txt > > > when a job has many input paths - listStatus - or the improved > listLocatedStatus - calls (invoked from the getSplits() method) can take a > long time. Most of the time is spent waiting for the previous call to > complete and then dispatching the next call. > This can be greatly speeded up by dispatching multiple calls at once (via > executors). If the same filesystem client is used - then the calls are much > better pipelined (since calls are serialized) and don't impose extra burden > on the namenode while at the same time greatly reducing the latency to the > client. In a simple test on non-peak hours, this resulted in the getSplits() > time reducing from about 3s to about 0.5s. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (MAPREDUCE-5793) Make TestMRJobsWithProfiler#testProfiler faster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA resolved MAPREDUCE-5793. -- Resolution: Duplicate Closing this issue as duplicate. I suppose the patch uploaded to MAPREDUCE-5804 will fix it. > Make TestMRJobsWithProfiler#testProfiler faster > --- > > Key: MAPREDUCE-5793 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5793 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Akira AJISAKA > > TestMRJobsWithProfiler#testProfiler sometimes took more than 120 seconds and > then got timeout. > https://builds.apache.org/job/PreCommit-HADOOP-Build/3656//testReport/org.apache.hadoop.mapreduce.v2/TestMRJobsWithProfiler/testProfiler/ -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5804) TestMRJobsWithProfiler#testProfiler timesout
[
https://issues.apache.org/jira/browse/MAPREDUCE-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941114#comment-13941114
]
Akira AJISAKA commented on MAPREDUCE-5804:
--
The timeout occurred in Jenkins and I filed MAPREDUCE-5793.
https://builds.apache.org/job/PreCommit-HADOOP-Build/3656//testReport/org.apache.hadoop.mapreduce.v2/TestMRJobsWithProfiler/testProfiler
I thought it would be better to get the test faster instead of simply
increasing the timeout, however, I'm okay to increase the timeout because now I
suppose it's difficult to make it faster.
I'll close MAPREDUCE-5793.
> TestMRJobsWithProfiler#testProfiler timesout
>
>
> Key: MAPREDUCE-5804
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5804
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Affects Versions: 2.4.0
>Reporter: Mit Desai
>Assignee: Mit Desai
> Attachments: LOG.txt, MAPREDUCE-5804.patch
>
>
> {noformat}
> testProfiler(org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler) Time
> elapsed: 154.972 sec <<< ERROR!
> java.lang.Exception: test timed out after 12 milliseconds
> at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
> at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
> at java.io.File.exists(File.java:813)
> at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1080)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:358)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at org.apache.log4j.spi.LoggingEvent.(LoggingEvent.java:165)
> at org.apache.log4j.Category.forcedLog(Category.java:391)
> at org.apache.log4j.Category.log(Category.java:856)
> at
> org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:208)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:338)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
> at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:532)
> at org.apache.hadoop.mapreduce.Job$1.run(Job.java:314)
> at org.apache.hadoop.mapreduce.Job$1.run(Job.java:311)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1570)
> at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:311)
> at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:599)
> at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1344)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1306)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testProfiler(TestMRJobsWithProfiler.java:138)
> Results :
> Tests in error:
> TestMRJobsWithProfiler.testProfiler:138 » test timed out after 12
> millise...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5804) TestMRJobsWithProfiler#testProfiler timesout
[
https://issues.apache.org/jira/browse/MAPREDUCE-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mit Desai updated MAPREDUCE-5804:
-
Attachment: MAPREDUCE-5804.patch
Attaching the patch for trunk and branch-2
> TestMRJobsWithProfiler#testProfiler timesout
>
>
> Key: MAPREDUCE-5804
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5804
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Affects Versions: 2.4.0
>Reporter: Mit Desai
>Assignee: Mit Desai
> Attachments: LOG.txt, MAPREDUCE-5804.patch
>
>
> {noformat}
> testProfiler(org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler) Time
> elapsed: 154.972 sec <<< ERROR!
> java.lang.Exception: test timed out after 12 milliseconds
> at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
> at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
> at java.io.File.exists(File.java:813)
> at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1080)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:358)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at org.apache.log4j.spi.LoggingEvent.(LoggingEvent.java:165)
> at org.apache.log4j.Category.forcedLog(Category.java:391)
> at org.apache.log4j.Category.log(Category.java:856)
> at
> org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:208)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:338)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
> at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:532)
> at org.apache.hadoop.mapreduce.Job$1.run(Job.java:314)
> at org.apache.hadoop.mapreduce.Job$1.run(Job.java:311)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1570)
> at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:311)
> at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:599)
> at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1344)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1306)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testProfiler(TestMRJobsWithProfiler.java:138)
> Results :
> Tests in error:
> TestMRJobsWithProfiler.testProfiler:138 » test timed out after 12
> millise...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5804) TestMRJobsWithProfiler#testProfiler timesout
[
https://issues.apache.org/jira/browse/MAPREDUCE-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mit Desai updated MAPREDUCE-5804:
-
Status: Patch Available (was: Open)
> TestMRJobsWithProfiler#testProfiler timesout
>
>
> Key: MAPREDUCE-5804
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5804
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Affects Versions: 2.4.0
>Reporter: Mit Desai
>Assignee: Mit Desai
> Attachments: LOG.txt, MAPREDUCE-5804.patch
>
>
> {noformat}
> testProfiler(org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler) Time
> elapsed: 154.972 sec <<< ERROR!
> java.lang.Exception: test timed out after 12 milliseconds
> at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
> at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
> at java.io.File.exists(File.java:813)
> at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1080)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:358)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at org.apache.log4j.spi.LoggingEvent.(LoggingEvent.java:165)
> at org.apache.log4j.Category.forcedLog(Category.java:391)
> at org.apache.log4j.Category.log(Category.java:856)
> at
> org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:208)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:338)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
> at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:532)
> at org.apache.hadoop.mapreduce.Job$1.run(Job.java:314)
> at org.apache.hadoop.mapreduce.Job$1.run(Job.java:311)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1570)
> at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:311)
> at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:599)
> at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1344)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1306)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testProfiler(TestMRJobsWithProfiler.java:138)
> Results :
> Tests in error:
> TestMRJobsWithProfiler.testProfiler:138 » test timed out after 12
> millise...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5804) TestMRJobsWithProfiler#testProfiler timesout
[
https://issues.apache.org/jira/browse/MAPREDUCE-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mit Desai updated MAPREDUCE-5804:
-
Attachment: LOG.txt
I have attached the log.txt that has the standard output from our nightly
build. I have tried reproducing the failure but I was only able to do that once
in many times. And lost the surefire-report as I was running the test in a loop.
I think the timeout is due to the machine load. I have a patch that I will be
uploading soon. It increases the timeout value so the test has enough time even
when tons of jobs are running.
> TestMRJobsWithProfiler#testProfiler timesout
>
>
> Key: MAPREDUCE-5804
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5804
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Affects Versions: 2.4.0
>Reporter: Mit Desai
>Assignee: Mit Desai
> Attachments: LOG.txt
>
>
> {noformat}
> testProfiler(org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler) Time
> elapsed: 154.972 sec <<< ERROR!
> java.lang.Exception: test timed out after 12 milliseconds
> at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
> at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
> at java.io.File.exists(File.java:813)
> at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1080)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:358)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at org.apache.log4j.spi.LoggingEvent.(LoggingEvent.java:165)
> at org.apache.log4j.Category.forcedLog(Category.java:391)
> at org.apache.log4j.Category.log(Category.java:856)
> at
> org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:208)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:338)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
> at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:532)
> at org.apache.hadoop.mapreduce.Job$1.run(Job.java:314)
> at org.apache.hadoop.mapreduce.Job$1.run(Job.java:311)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1570)
> at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:311)
> at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:599)
> at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1344)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1306)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testProfiler(TestMRJobsWithProfiler.java:138)
> Results :
> Tests in error:
> TestMRJobsWithProfiler.testProfiler:138 » test timed out after 12
> millise...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (MAPREDUCE-5804) TestMRJobsWithProfiler#testProfiler timesout
[
https://issues.apache.org/jira/browse/MAPREDUCE-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mit Desai reassigned MAPREDUCE-5804:
Assignee: Mit Desai
> TestMRJobsWithProfiler#testProfiler timesout
>
>
> Key: MAPREDUCE-5804
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5804
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Affects Versions: 2.4.0
>Reporter: Mit Desai
>Assignee: Mit Desai
>
> {noformat}
> testProfiler(org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler) Time
> elapsed: 154.972 sec <<< ERROR!
> java.lang.Exception: test timed out after 12 milliseconds
> at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
> at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
> at java.io.File.exists(File.java:813)
> at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1080)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:358)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at org.apache.log4j.spi.LoggingEvent.(LoggingEvent.java:165)
> at org.apache.log4j.Category.forcedLog(Category.java:391)
> at org.apache.log4j.Category.log(Category.java:856)
> at
> org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:208)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:338)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
> at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:532)
> at org.apache.hadoop.mapreduce.Job$1.run(Job.java:314)
> at org.apache.hadoop.mapreduce.Job$1.run(Job.java:311)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1570)
> at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:311)
> at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:599)
> at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1344)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1306)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testProfiler(TestMRJobsWithProfiler.java:138)
> Results :
> Tests in error:
> TestMRJobsWithProfiler.testProfiler:138 » test timed out after 12
> millise...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5804) TestMRJobsWithProfiler#testProfiler timesout
[
https://issues.apache.org/jira/browse/MAPREDUCE-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941012#comment-13941012
]
Gera Shegalov commented on MAPREDUCE-5804:
--
[~mdesai], Is there some statistics showing that this test is flapping? It does
not show up in our builds.
> TestMRJobsWithProfiler#testProfiler timesout
>
>
> Key: MAPREDUCE-5804
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5804
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Affects Versions: 2.4.0
>Reporter: Mit Desai
>
> {noformat}
> testProfiler(org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler) Time
> elapsed: 154.972 sec <<< ERROR!
> java.lang.Exception: test timed out after 12 milliseconds
> at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
> at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
> at java.io.File.exists(File.java:813)
> at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1080)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:358)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at org.apache.log4j.spi.LoggingEvent.(LoggingEvent.java:165)
> at org.apache.log4j.Category.forcedLog(Category.java:391)
> at org.apache.log4j.Category.log(Category.java:856)
> at
> org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:208)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:338)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
> at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:532)
> at org.apache.hadoop.mapreduce.Job$1.run(Job.java:314)
> at org.apache.hadoop.mapreduce.Job$1.run(Job.java:311)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1570)
> at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:311)
> at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:599)
> at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1344)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1306)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testProfiler(TestMRJobsWithProfiler.java:138)
> Results :
> Tests in error:
> TestMRJobsWithProfiler.testProfiler:138 » test timed out after 12
> millise...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-2349) speed up list[located]status calls from input formats
[ https://issues.apache.org/jira/browse/MAPREDUCE-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-2349: --- Status: Open (was: Patch Available) Started looking at it, nicely done patch! Few minor comments - Add the configs to mapred-default.xml as documentation? - LIST_STATUS_NUM_THREADS_DEFAULT -> DEFAULT_LIST_STATUS_NUM_THREADS - oldListStatus() -> singleThreadedListStatus() - Can you add a bit of javadoc to all the new classes and methods in LocatedFileStatusFetcher? Also to the main LocatedFileStatusFetcher class itself. - Synchronization needed for ProcessInitialInputPathResult.addError()? - Can you group the callable, result and call-back for each type of operation together in two classes? - The 'result' variable doesn't need to be a class field of ProcessInputDirCallable. Similarly the one in ProcessInitialInputPathCallable. The tests look good! > speed up list[located]status calls from input formats > - > > Key: MAPREDUCE-2349 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2349 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Reporter: Joydeep Sen Sarma >Assignee: Siddharth Seth > Attachments: MAPREDUCE-2349.1.wip.txt, MAPREDUCE-2349.2.txt, > MAPREDUCE-2349.3.txt, MAPREDUCE-2349.4.txt > > > when a job has many input paths - listStatus - or the improved > listLocatedStatus - calls (invoked from the getSplits() method) can take a > long time. Most of the time is spent waiting for the previous call to > complete and then dispatching the next call. > This can be greatly speeded up by dispatching multiple calls at once (via > executors). If the same filesystem client is used - then the calls are much > better pipelined (since calls are serialized) and don't impose extra burden > on the namenode while at the same time greatly reducing the latency to the > client. In a simple test on non-peak hours, this resulted in the getSplits() > time reducing from about 3s to about 0.5s. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5748) Potential null pointer deference in ShuffleHandler#Shuffle#messageReceived()
[
https://issues.apache.org/jira/browse/MAPREDUCE-5748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13940969#comment-13940969
]
Ted Yu commented on MAPREDUCE-5748:
---
+1 if tests pass.
Thanks
> Potential null pointer deference in ShuffleHandler#Shuffle#messageReceived()
>
>
> Key: MAPREDUCE-5748
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5748
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
> Attachments:
> 0001-MAPREDUCE-5748-Potential-null-pointer-deference-in-S.patch
>
>
> Starting around line 510:
> {code}
> ChannelFuture lastMap = null;
> for (String mapId : mapIds) {
> ...
> }
> lastMap.addListener(metrics);
> lastMap.addListener(ChannelFutureListener.CLOSE);
> {code}
> If mapIds is empty, lastMap would remain null, leading to NPE in
> addListener() call.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5748) Potential null pointer deference in ShuffleHandler#Shuffle#messageReceived()
[
https://issues.apache.org/jira/browse/MAPREDUCE-5748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hussein Baghdadi updated MAPREDUCE-5748:
Attachment: 0001-MAPREDUCE-5748-Potential-null-pointer-deference-in-S.patch
> Potential null pointer deference in ShuffleHandler#Shuffle#messageReceived()
>
>
> Key: MAPREDUCE-5748
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5748
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
> Attachments:
> 0001-MAPREDUCE-5748-Potential-null-pointer-deference-in-S.patch
>
>
> Starting around line 510:
> {code}
> ChannelFuture lastMap = null;
> for (String mapId : mapIds) {
> ...
> }
> lastMap.addListener(metrics);
> lastMap.addListener(ChannelFutureListener.CLOSE);
> {code}
> If mapIds is empty, lastMap would remain null, leading to NPE in
> addListener() call.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5748) Potential null pointer deference in ShuffleHandler#Shuffle#messageReceived()
[
https://issues.apache.org/jira/browse/MAPREDUCE-5748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13940951#comment-13940951
]
Hussein Baghdadi commented on MAPREDUCE-5748:
-
I tried to fix it. Kindly check the attached patched file.
> Potential null pointer deference in ShuffleHandler#Shuffle#messageReceived()
>
>
> Key: MAPREDUCE-5748
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5748
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
> Attachments:
> 0001-MAPREDUCE-5748-Potential-null-pointer-deference-in-S.patch
>
>
> Starting around line 510:
> {code}
> ChannelFuture lastMap = null;
> for (String mapId : mapIds) {
> ...
> }
> lastMap.addListener(metrics);
> lastMap.addListener(ChannelFutureListener.CLOSE);
> {code}
> If mapIds is empty, lastMap would remain null, leading to NPE in
> addListener() call.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5805) Unable to parse launch time from job history file
[
https://issues.apache.org/jira/browse/MAPREDUCE-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13940859#comment-13940859
]
Hadoop QA commented on MAPREDUCE-5805:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12635609/MAPREDUCE-5805.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4443//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4443//console
This message is automatically generated.
> Unable to parse launch time from job history file
> -
>
> Key: MAPREDUCE-5805
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5805
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.3.0
>Reporter: Fengdong Yu
>Assignee: Akira AJISAKA
> Attachments: MAPREDUCE-5805.patch
>
>
> when job complete, there are WARN complains in the log:
> {code}
> 2014-03-19 13:31:10,036 WARN
> org.apache.hadoop.mapreduce.v2.jobhistory.FileNameIndexUtils: Unable to parse
> launch time from job history file
> job_1395204058904_0003-1395206473646-root-test_one_word-1395206966214-4-2-SUCCEEDED-root.test-queue-1395206480070.jhist
> : java.lang.NumberFormatException: For input string: "queue"
> {code}
> because there is (-) in the queue name 'test-queue', we split the job
> history file name by (-), and get the ninth item as job start time.
> FileNameIndexUtils.java
> {code}
> private static final int JOB_START_TIME_INDEX = 9;
> {code}
> but there is another potential issue:
> if I also include '-' in the job name(test_one_world in this case), there are
> all misunderstand.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5805) Unable to parse launch time from job history file
[
https://issues.apache.org/jira/browse/MAPREDUCE-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated MAPREDUCE-5805:
-
Attachment: MAPREDUCE-5805.patch
> Unable to parse launch time from job history file
> -
>
> Key: MAPREDUCE-5805
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5805
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.3.0
>Reporter: Fengdong Yu
>Assignee: Akira AJISAKA
>Priority: Minor
> Attachments: MAPREDUCE-5805.patch
>
>
> when job complete, there are WARN complains in the log:
> {code}
> 2014-03-19 13:31:10,036 WARN
> org.apache.hadoop.mapreduce.v2.jobhistory.FileNameIndexUtils: Unable to parse
> launch time from job history file
> job_1395204058904_0003-1395206473646-root-test_one_word-1395206966214-4-2-SUCCEEDED-root.test-queue-1395206480070.jhist
> : java.lang.NumberFormatException: For input string: "queue"
> {code}
> because there is (-) in the queue name 'test-queue', we split the job
> history file name by (-), and get the ninth item as job start time.
> FileNameIndexUtils.java
> {code}
> private static final int JOB_START_TIME_INDEX = 9;
> {code}
> but there is another potential issue:
> if I also include '-' in the job name(test_one_world in this case), there are
> all misunderstand.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5805) Unable to parse launch time from job history file
[
https://issues.apache.org/jira/browse/MAPREDUCE-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated MAPREDUCE-5805:
-
Target Version/s: 2.4.0
Status: Patch Available (was: Open)
> Unable to parse launch time from job history file
> -
>
> Key: MAPREDUCE-5805
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5805
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.3.0
>Reporter: Fengdong Yu
>Assignee: Akira AJISAKA
>Priority: Minor
> Attachments: MAPREDUCE-5805.patch
>
>
> when job complete, there are WARN complains in the log:
> {code}
> 2014-03-19 13:31:10,036 WARN
> org.apache.hadoop.mapreduce.v2.jobhistory.FileNameIndexUtils: Unable to parse
> launch time from job history file
> job_1395204058904_0003-1395206473646-root-test_one_word-1395206966214-4-2-SUCCEEDED-root.test-queue-1395206480070.jhist
> : java.lang.NumberFormatException: For input string: "queue"
> {code}
> because there is (-) in the queue name 'test-queue', we split the job
> history file name by (-), and get the ninth item as job start time.
> FileNameIndexUtils.java
> {code}
> private static final int JOB_START_TIME_INDEX = 9;
> {code}
> but there is another potential issue:
> if I also include '-' in the job name(test_one_world in this case), there are
> all misunderstand.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5805) Unable to parse launch time from job history file
[
https://issues.apache.org/jira/browse/MAPREDUCE-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated MAPREDUCE-5805:
-
Priority: Major (was: Minor)
> Unable to parse launch time from job history file
> -
>
> Key: MAPREDUCE-5805
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5805
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.3.0
>Reporter: Fengdong Yu
>Assignee: Akira AJISAKA
> Attachments: MAPREDUCE-5805.patch
>
>
> when job complete, there are WARN complains in the log:
> {code}
> 2014-03-19 13:31:10,036 WARN
> org.apache.hadoop.mapreduce.v2.jobhistory.FileNameIndexUtils: Unable to parse
> launch time from job history file
> job_1395204058904_0003-1395206473646-root-test_one_word-1395206966214-4-2-SUCCEEDED-root.test-queue-1395206480070.jhist
> : java.lang.NumberFormatException: For input string: "queue"
> {code}
> because there is (-) in the queue name 'test-queue', we split the job
> history file name by (-), and get the ninth item as job start time.
> FileNameIndexUtils.java
> {code}
> private static final int JOB_START_TIME_INDEX = 9;
> {code}
> but there is another potential issue:
> if I also include '-' in the job name(test_one_world in this case), there are
> all misunderstand.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Moved] (MAPREDUCE-5805) Unable to parse launch time from job history file
[
https://issues.apache.org/jira/browse/MAPREDUCE-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA moved YARN-1851 to MAPREDUCE-5805:
Component/s: (was: resourcemanager)
jobhistoryserver
Fix Version/s: (was: 2.4.0)
Affects Version/s: (was: 2.4.0)
2.3.0
Key: MAPREDUCE-5805 (was: YARN-1851)
Project: Hadoop Map/Reduce (was: Hadoop YARN)
> Unable to parse launch time from job history file
> -
>
> Key: MAPREDUCE-5805
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5805
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.3.0
>Reporter: Fengdong Yu
>Assignee: Akira AJISAKA
>Priority: Minor
>
> when job complete, there are WARN complains in the log:
> {code}
> 2014-03-19 13:31:10,036 WARN
> org.apache.hadoop.mapreduce.v2.jobhistory.FileNameIndexUtils: Unable to parse
> launch time from job history file
> job_1395204058904_0003-1395206473646-root-test_one_word-1395206966214-4-2-SUCCEEDED-root.test-queue-1395206480070.jhist
> : java.lang.NumberFormatException: For input string: "queue"
> {code}
> because there is (-) in the queue name 'test-queue', we split the job
> history file name by (-), and get the ninth item as job start time.
> FileNameIndexUtils.java
> {code}
> private static final int JOB_START_TIME_INDEX = 9;
> {code}
> but there is another potential issue:
> if I also include '-' in the job name(test_one_world in this case), there are
> all misunderstand.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5805) Unable to parse launch time from job history file
[
https://issues.apache.org/jira/browse/MAPREDUCE-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13940784#comment-13940784
]
Akira AJISAKA commented on MAPREDUCE-5805:
--
Moved to MapReduce project.
> Unable to parse launch time from job history file
> -
>
> Key: MAPREDUCE-5805
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5805
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.3.0
>Reporter: Fengdong Yu
>Assignee: Akira AJISAKA
>Priority: Minor
>
> when job complete, there are WARN complains in the log:
> {code}
> 2014-03-19 13:31:10,036 WARN
> org.apache.hadoop.mapreduce.v2.jobhistory.FileNameIndexUtils: Unable to parse
> launch time from job history file
> job_1395204058904_0003-1395206473646-root-test_one_word-1395206966214-4-2-SUCCEEDED-root.test-queue-1395206480070.jhist
> : java.lang.NumberFormatException: For input string: "queue"
> {code}
> because there is (-) in the queue name 'test-queue', we split the job
> history file name by (-), and get the ninth item as job start time.
> FileNameIndexUtils.java
> {code}
> private static final int JOB_START_TIME_INDEX = 9;
> {code}
> but there is another potential issue:
> if I also include '-' in the job name(test_one_world in this case), there are
> all misunderstand.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (MAPREDUCE-5804) TestMRJobsWithProfiler#testProfiler timesout
Mit Desai created MAPREDUCE-5804:
Summary: TestMRJobsWithProfiler#testProfiler timesout
Key: MAPREDUCE-5804
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5804
Project: Hadoop Map/Reduce
Issue Type: Test
Affects Versions: 2.4.0
Reporter: Mit Desai
{noformat}
testProfiler(org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler) Time
elapsed: 154.972 sec <<< ERROR!
java.lang.Exception: test timed out after 12 milliseconds
at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
at java.io.File.exists(File.java:813)
at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1080)
at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
at java.net.URLClassLoader$1.run(URLClassLoader.java:358)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.apache.log4j.spi.LoggingEvent.(LoggingEvent.java:165)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
at
org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:208)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:338)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:532)
at org.apache.hadoop.mapreduce.Job$1.run(Job.java:314)
at org.apache.hadoop.mapreduce.Job$1.run(Job.java:311)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1570)
at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:311)
at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:599)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1344)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1306)
at
org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testProfiler(TestMRJobsWithProfiler.java:138)
Results :
Tests in error:
TestMRJobsWithProfiler.testProfiler:138 » test timed out after 12
millise...
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
