[jira] [Commented] (MAPREDUCE-7334) TestJobEndNotifier fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17317724#comment-17317724
]
Jing Zhao commented on MAPREDUCE-7334:
--
+1. I've merged the PR.
> TestJobEndNotifier fails
>
>
> Key: MAPREDUCE-7334
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7334
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/8/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt]
> {quote}
> [INFO] Running org.apache.hadoop.mapred.TestJobEndNotifier
> [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 47.81
> s <<< FAILURE! - in org.apache.hadoop.mapred.TestJobEndNotifier
> [ERROR] testNotificationTimeout(org.apache.hadoop.mapred.TestJobEndNotifier)
> Time elapsed: 47.322 s <<< ERROR!
> java.util.concurrent.TimeoutException
> at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:130)
> at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:30)
> at
> org.eclipse.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:175)
> at org.eclipse.jetty.server.Server.doStop(Server.java:447)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94)
> at org.apache.hadoop.http.HttpServer2.stop(HttpServer2.java:1499)
> at
> org.apache.hadoop.mapred.TestJobEndNotifier.tearDown(TestJobEndNotifier.java:127)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {quote}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6248) Allow users to get the MR job information for distcp
[
https://issues.apache.org/jira/browse/MAPREDUCE-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346020#comment-14346020
]
Jing Zhao commented on MAPREDUCE-6248:
--
The eclipse:eclipse failure should be unrelated and it worked in my local
machine.
> Allow users to get the MR job information for distcp
>
>
> Key: MAPREDUCE-6248
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6248
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distcp
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: MAPREDUCE-6248.000.patch, MAPREDUCE-6248.001.patch
>
>
> Currently the DistCp is acting as a tool and the corresponding MapReduce Job
> is created and used inside of its {{execute}} method. It is thus difficult
> for external services to query its progress and counters. It may be helpful
> to persist the job id into a file inside its staging directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6248) Allow users to get the MR job information for distcp
[
https://issues.apache.org/jira/browse/MAPREDUCE-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated MAPREDUCE-6248:
-
Attachment: MAPREDUCE-6248.001.patch
Thanks for the review, Vinod! Update the patch to address your comments. Now
DistCp keeps its constructor, {{run}} (overriding {{Tool}}), and {{execute}}
(which has been called by downstream applications) as public. Also two
additional public methods are added: {{createAndSubmitJob}} and
{{waitForJobCompletion}}.
> Allow users to get the MR job information for distcp
>
>
> Key: MAPREDUCE-6248
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6248
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distcp
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: MAPREDUCE-6248.000.patch, MAPREDUCE-6248.001.patch
>
>
> Currently the DistCp is acting as a tool and the corresponding MapReduce Job
> is created and used inside of its {{execute}} method. It is thus difficult
> for external services to query its progress and counters. It may be helpful
> to persist the job id into a file inside its staging directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6248) Allow users to get the MR job information for distcp
[
https://issues.apache.org/jira/browse/MAPREDUCE-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated MAPREDUCE-6248:
-
Status: Patch Available (was: Open)
> Allow users to get the MR job information for distcp
>
>
> Key: MAPREDUCE-6248
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6248
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distcp
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: MAPREDUCE-6248.000.patch
>
>
> Currently the DistCp is acting as a tool and the corresponding MapReduce Job
> is created and used inside of its {{execute}} method. It is thus difficult
> for external services to query its progress and counters. It may be helpful
> to persist the job id into a file inside its staging directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6248) Allow users to get the MR job information for distcp
[
https://issues.apache.org/jira/browse/MAPREDUCE-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated MAPREDUCE-6248:
-
Summary: Allow users to get the MR job information for distcp (was:
Persist DistCp job id in the staging directory)
> Allow users to get the MR job information for distcp
>
>
> Key: MAPREDUCE-6248
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6248
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distcp
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: MAPREDUCE-6248.000.patch
>
>
> Currently the DistCp is acting as a tool and the corresponding MapReduce Job
> is created and used inside of its {{execute}} method. It is thus difficult
> for external services to query its progress and counters. It may be helpful
> to persist the job id into a file inside its staging directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6248) Persist DistCp job id in the staging directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated MAPREDUCE-6248:
-
Attachment: MAPREDUCE-6248.000.patch
Patch implementing [~vinodkv]'s proposal. The patch simply adds a new public
{{createAndSubmitJob}} method which returns the mapreduce job object.
> Persist DistCp job id in the staging directory
> --
>
> Key: MAPREDUCE-6248
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6248
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distcp
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: MAPREDUCE-6248.000.patch
>
>
> Currently the DistCp is acting as a tool and the corresponding MapReduce Job
> is created and used inside of its {{execute}} method. It is thus difficult
> for external services to query its progress and counters. It may be helpful
> to persist the job id into a file inside its staging directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6248) Persist DistCp job id in the staging directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313414#comment-14313414
]
Jing Zhao commented on MAPREDUCE-6248:
--
yes, actually that will be even better! I will upload a patch for this later.
> Persist DistCp job id in the staging directory
> --
>
> Key: MAPREDUCE-6248
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6248
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distcp
>Reporter: Jing Zhao
>Assignee: Jing Zhao
>
> Currently the DistCp is acting as a tool and the corresponding MapReduce Job
> is created and used inside of its {{execute}} method. It is thus difficult
> for external services to query its progress and counters. It may be helpful
> to persist the job id into a file inside its staging directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6248) Persist DistCp job id in the staging directory
Jing Zhao created MAPREDUCE-6248:
Summary: Persist DistCp job id in the staging directory
Key: MAPREDUCE-6248
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6248
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: distcp
Reporter: Jing Zhao
Assignee: Jing Zhao
Currently the DistCp is acting as a tool and the corresponding MapReduce Job
is created and used inside of its {{execute}} method. It is thus difficult for
external services to query its progress and counters. It may be helpful to
persist the job id into a file inside its staging directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5899) Support incremental data copy in DistCp
[ https://issues.apache.org/jira/browse/MAPREDUCE-5899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated MAPREDUCE-5899: - Resolution: Fixed Fix Version/s: 2.5.0 Status: Resolved (was: Patch Available) I've committed this to trunk and branch-2. > Support incremental data copy in DistCp > --- > > Key: MAPREDUCE-5899 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5899 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.5.0 > > Attachments: HADOOP-10608.000.patch, HADOOP-10608.001.patch, > MAPREDUCE-5899.002.patch, MAPREDUCE-5899.002.patch > > > Currently when doing distcp with -update option, for two files with the same > file names but with different file length or checksum, we overwrite the whole > file. It will be good if we can detect the case where (sourceFile = > targetFile + appended_data), and only transfer the appended data segment to > the target. This will be very useful if we're doing incremental distcp. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5899) Support incremental data copy in DistCp
[ https://issues.apache.org/jira/browse/MAPREDUCE-5899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14006229#comment-14006229 ] Jing Zhao commented on MAPREDUCE-5899: -- The failed unit test should be unrelated. It's also seen in recent Jenkins run like MAPREDUCE-5809. I will commit the patch shortly. > Support incremental data copy in DistCp > --- > > Key: MAPREDUCE-5899 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5899 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HADOOP-10608.000.patch, HADOOP-10608.001.patch, > MAPREDUCE-5899.002.patch, MAPREDUCE-5899.002.patch > > > Currently when doing distcp with -update option, for two files with the same > file names but with different file length or checksum, we overwrite the whole > file. It will be good if we can detect the case where (sourceFile = > targetFile + appended_data), and only transfer the appended data segment to > the target. This will be very useful if we're doing incremental distcp. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5899) Support incremental data copy in DistCp
[ https://issues.apache.org/jira/browse/MAPREDUCE-5899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated MAPREDUCE-5899: - Status: Patch Available (was: Open) > Support incremental data copy in DistCp > --- > > Key: MAPREDUCE-5899 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5899 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HADOOP-10608.000.patch, HADOOP-10608.001.patch, > MAPREDUCE-5899.002.patch, MAPREDUCE-5899.002.patch > > > Currently when doing distcp with -update option, for two files with the same > file names but with different file length or checksum, we overwrite the whole > file. It will be good if we can detect the case where (sourceFile = > targetFile + appended_data), and only transfer the appended data segment to > the target. This will be very useful if we're doing incremental distcp. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5899) Support incremental data copy in DistCp
[ https://issues.apache.org/jira/browse/MAPREDUCE-5899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated MAPREDUCE-5899: - Status: Open (was: Patch Available) > Support incremental data copy in DistCp > --- > > Key: MAPREDUCE-5899 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5899 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HADOOP-10608.000.patch, HADOOP-10608.001.patch, > MAPREDUCE-5899.002.patch, MAPREDUCE-5899.002.patch > > > Currently when doing distcp with -update option, for two files with the same > file names but with different file length or checksum, we overwrite the whole > file. It will be good if we can detect the case where (sourceFile = > targetFile + appended_data), and only transfer the appended data segment to > the target. This will be very useful if we're doing incremental distcp. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5899) Support incremental data copy in DistCp
[ https://issues.apache.org/jira/browse/MAPREDUCE-5899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated MAPREDUCE-5899: - Attachment: MAPREDUCE-5899.002.patch Resubmit the patch to trigger Jenkins. > Support incremental data copy in DistCp > --- > > Key: MAPREDUCE-5899 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5899 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HADOOP-10608.000.patch, HADOOP-10608.001.patch, > MAPREDUCE-5899.002.patch, MAPREDUCE-5899.002.patch > > > Currently when doing distcp with -update option, for two files with the same > file names but with different file length or checksum, we overwrite the whole > file. It will be good if we can detect the case where (sourceFile = > targetFile + appended_data), and only transfer the appended data segment to > the target. This will be very useful if we're doing incremental distcp. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5899) Support incremental data copy in DistCp
[ https://issues.apache.org/jira/browse/MAPREDUCE-5899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated MAPREDUCE-5899: - Attachment: MAPREDUCE-5899.002.patch Thanks again, Nicholas! Update the patch to fix javac/findbug warnings and unit tests. > Support incremental data copy in DistCp > --- > > Key: MAPREDUCE-5899 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5899 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HADOOP-10608.000.patch, HADOOP-10608.001.patch, > MAPREDUCE-5899.002.patch > > > Currently when doing distcp with -update option, for two files with the same > file names but with different file length or checksum, we overwrite the whole > file. It will be good if we can detect the case where (sourceFile = > targetFile + appended_data), and only transfer the appended data segment to > the target. This will be very useful if we're doing incremental distcp. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-4779) Unit test TestJobTrackerSafeMode fails with ant 1.8.3+
[ https://issues.apache.org/jira/browse/MAPREDUCE-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496510#comment-13496510 ] Jing Zhao commented on MAPREDUCE-4779: -- Amir, the patch submission and automatic building/testing is only used by patch for trunk. Thus you may want to cancel your patch and run ant test and testpatch locally. Also I tried to apply your patch to branch-1. Seems you need to rebase the patch. > Unit test TestJobTrackerSafeMode fails with ant 1.8.3+ > --- > > Key: MAPREDUCE-4779 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4779 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 1.0.3 > Environment: Fedora 17 on x86_64 >Reporter: Amir Sanjar >Assignee: Amir Sanjar > Attachments: MAPREDUCE-4779-branch-1.patch > > > Problem: > JUnit tag @Ignore is not recognized since the testcase is JUnit3 and not > JUnit4: > Solution: > Migrate the testcase to JUnit4 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
