[jira] [Commented] (MAPREDUCE-6740) Enforce mapreduce.task.timeout to be at least mapreduce.task.progress-report.interval
[
https://issues.apache.org/jira/browse/MAPREDUCE-6740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15423980#comment-15423980
]
Hadoop QA commented on MAPREDUCE-6740:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 36s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 4s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 45s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} hadoop-mapreduce-project/hadoop-mapreduce-client:
The patch generated 0 new + 723 unchanged - 3 fixed = 723 total (was 726)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 18s
{color} | {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core
generated 2 new + 2508 unchanged - 0 fixed = 2510 total (was 2508) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 13s {color}
| {color:red} hadoop-mapreduce-client-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 47s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 23s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapred.TestTaskProgressReporter |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824079/mapreduce6740.005.patch
|
| JIRA Issue | MAPREDUCE-6740 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 78e144d1b676 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2353271 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| javadoc |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6674/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
|
[jira] [Updated] (MAPREDUCE-6740) Enforce mapreduce.task.timeout to be at least mapreduce.task.progress-report.interval
[ https://issues.apache.org/jira/browse/MAPREDUCE-6740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6740: -- Attachment: mapreduce6740.005.patch Per offline discussion with Karthik, the default value of task progress report interval can be set to a portion of the value of task timeout. This gets rid of the default value of task progress interval in MRJobConfig.java. In this patch, the ratio is set to 0.01 so that in cases where both task report interval and task timeout are not set explicitly, the value of task progress report defaults to 3000 milliseconds, preserving its current default value. > Enforce mapreduce.task.timeout to be at least > mapreduce.task.progress-report.interval > - > > Key: MAPREDUCE-6740 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6740 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.8.0 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Attachments: mapreduce6740.001.patch, mapreduce6740.002.patch, > mapreduce6740.003.patch, mapreduce6740.004.patch, mapreduce6740.005.patch > > > Mapreduce-6242 makes task status update interval configurable to ease the > pressure on MR AM to process status updates, but it did not ensure that > mapreduce.task.timeout is no smaller than the configured value of task report > interval. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6690) Limit the number of resources a single map reduce job can submit for localization
[
https://issues.apache.org/jira/browse/MAPREDUCE-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15423747#comment-15423747
]
Hadoop QA commented on MAPREDUCE-6690:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 35s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 11s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 118m 49s
{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 140m 41s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824023/MAPREDUCE-6690-trunk-v7.patch
|
| JIRA Issue | MAPREDUCE-6690 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 2bad52d01f13 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 27a6e09 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6673/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output |
h
[jira] [Created] (MAPREDUCE-6758) TestDFSIO should parallelize its creation of control files on setup
Dennis Huo created MAPREDUCE-6758:
-
Summary: TestDFSIO should parallelize its creation of control
files on setup
Key: MAPREDUCE-6758
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6758
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: test
Reporter: Dennis Huo
TestDFSIO currently performs a sequential for-loop to create {{nrFiles}}
control files in the {{controlDir}} which is a subdirectory of the overall
{{test.build.data}} directory, which may be a non-HDFS FileSystem
implementation:
{code:java}
private void createControlFile(FileSystem fs,
long nrBytes, // in bytes
int nrFiles
) throws IOException {
LOG.info("creating control file: "+nrBytes+" bytes, "+nrFiles+" files");
Path controlDir = getControlDir(config);
fs.delete(controlDir, true);
for(int i=0; i < nrFiles; i++) {
String name = getFileName(i);
Path controlFile = new Path(controlDir, "in_file_" + name);
SequenceFile.Writer writer = null;
try {
writer = SequenceFile.createWriter(fs, config, controlFile,
Text.class, LongWritable.class,
CompressionType.NONE);
writer.append(new Text(name), new LongWritable(nrBytes));
} catch(Exception e) {
throw new IOException(e.getLocalizedMessage());
} finally {
if (writer != null)
writer.close();
writer = null;
}
}
LOG.info("created control files for: "+nrFiles+" files");
}
{code}
When testing in an object-store based filesystem with higher round-trip latency
than HDFS (like S3 or GCS), this means job setup that might only take seconds
in HDFS ends up taking minutes or even tens of minutes against the object
stores if the test is using thousands of control files. In the same vein as
other JIRAs in [https://issues.apache.org/jira/browse/HADOOP-11694], the
control-file creation should be parallelized/multithreaded to efficiently
launch large TestDFSIO jobs against FileSystem impls with high round-trip
latency but which can still support high overall throughput/QPS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6690) Limit the number of resources a single map reduce job can submit for localization
[ https://issues.apache.org/jira/browse/MAPREDUCE-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15423593#comment-15423593 ] Chris Trezzo commented on MAPREDUCE-6690: - Thanks for the review [~jlowe]! Attached is a v7 patch. Here are the major changes: # Changes to address your comments around getStringCollection, totalConfigSize* and ensuring tests failed in the intended way. # Changes to make the usage of the word resource vs file consistent throughout the patch (i.e. a file is a type of resource). > Limit the number of resources a single map reduce job can submit for > localization > - > > Key: MAPREDUCE-6690 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6690 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Attachments: MAPREDUCE-6690-trunk-v1.patch, > MAPREDUCE-6690-trunk-v2.patch, MAPREDUCE-6690-trunk-v3.patch, > MAPREDUCE-6690-trunk-v4.patch, MAPREDUCE-6690-trunk-v5.patch, > MAPREDUCE-6690-trunk-v6.patch, MAPREDUCE-6690-trunk-v7.patch > > > Users will sometimes submit a large amount of resources to be localized as > part of a single map reduce job. This can cause issues with YARN localization > that destabilize the cluster and potentially impact other user jobs. These > resources are specified via the files, libjars, archives and jobjar command > line arguments or directly through the configuration (i.e. distributed cache > api). The resources specified could be too large in multiple dimensions: > # Total size > # Number of files > # Size of an individual resource (i.e. a large fat jar) > We would like to encourage good behavior on the client side by having the > option of enforcing resource limits along the above dimensions. > There should be a separate effort to enforce limits at the YARN layer on the > server side, but this jira is only covering the map reduce layer on the > client side. In practice, having these client side limits will get us a long > way towards preventing these localization anti-patterns. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6690) Limit the number of resources a single map reduce job can submit for localization
[ https://issues.apache.org/jira/browse/MAPREDUCE-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Trezzo updated MAPREDUCE-6690: Attachment: MAPREDUCE-6690-trunk-v7.patch > Limit the number of resources a single map reduce job can submit for > localization > - > > Key: MAPREDUCE-6690 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6690 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Attachments: MAPREDUCE-6690-trunk-v1.patch, > MAPREDUCE-6690-trunk-v2.patch, MAPREDUCE-6690-trunk-v3.patch, > MAPREDUCE-6690-trunk-v4.patch, MAPREDUCE-6690-trunk-v5.patch, > MAPREDUCE-6690-trunk-v6.patch, MAPREDUCE-6690-trunk-v7.patch > > > Users will sometimes submit a large amount of resources to be localized as > part of a single map reduce job. This can cause issues with YARN localization > that destabilize the cluster and potentially impact other user jobs. These > resources are specified via the files, libjars, archives and jobjar command > line arguments or directly through the configuration (i.e. distributed cache > api). The resources specified could be too large in multiple dimensions: > # Total size > # Number of files > # Size of an individual resource (i.e. a large fat jar) > We would like to encourage good behavior on the client side by having the > option of enforcing resource limits along the above dimensions. > There should be a separate effort to enforce limits at the YARN layer on the > server side, but this jira is only covering the map reduce layer on the > client side. In practice, having these client side limits will get us a long > way towards preventing these localization anti-patterns. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-6723) Turn log level to Debug in test
[ https://issues.apache.org/jira/browse/MAPREDUCE-6723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen resolved MAPREDUCE-6723. --- Resolution: Won't Fix > Turn log level to Debug in test > --- > > Key: MAPREDUCE-6723 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6723 > Project: Hadoop Map/Reduce > Issue Type: Test >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6723.001.patch > > > The current log level in test enviroment for all mapreduce projects is info. > Often in case where we are investigating intermittent test failures, DEBUG > level messages in log file can be very useful to identify problems. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6751) Add debug log message when splitting is not possible due to unsplittable compression
[ https://issues.apache.org/jira/browse/MAPREDUCE-6751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15423277#comment-15423277 ] Hudson commented on MAPREDUCE-6751: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10285 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10285/]) MAPREDUCE-6751. Add debug log message when splitting is not possible due (rchiang: rev 6c154abd33279475315b5f7f78dc47f1b0aa7028) * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java > Add debug log message when splitting is not possible due to unsplittable > compression > > > Key: MAPREDUCE-6751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6751 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client, mrv1, mrv2 >Affects Versions: 3.0.0-alpha1 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Labels: supportability > Fix For: 2.9.0 > > Attachments: MAPREDUCE-6751.2.patch, MAPREDUCE-6751.patch > > > There should be a message logged in case of the mapreduce will only spam one > mapper since the source file is compressed with an unsplitable algorithm -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6751) Add debug log message when splitting is not possible due to unsplittable compression
[ https://issues.apache.org/jira/browse/MAPREDUCE-6751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated MAPREDUCE-6751: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.9.0 Status: Resolved (was: Patch Available) Committed to trunk and branch-2. Thanks [~pvary] for the contribution! Thanks [~haibochen] for reviewing! > Add debug log message when splitting is not possible due to unsplittable > compression > > > Key: MAPREDUCE-6751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6751 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client, mrv1, mrv2 >Affects Versions: 3.0.0-alpha1 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Labels: supportability > Fix For: 2.9.0 > > Attachments: MAPREDUCE-6751.2.patch, MAPREDUCE-6751.patch > > > There should be a message logged in case of the mapreduce will only spam one > mapper since the source file is compressed with an unsplitable algorithm -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6751) Add debug log message when splitting is not possible due to unsplittable compression
[ https://issues.apache.org/jira/browse/MAPREDUCE-6751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated MAPREDUCE-6751: -- Summary: Add debug log message when splitting is not possible due to unsplittable compression (was: Add debug log message when splitting is not possible because of an unsplittable compression) > Add debug log message when splitting is not possible due to unsplittable > compression > > > Key: MAPREDUCE-6751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6751 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client, mrv1, mrv2 >Affects Versions: 3.0.0-alpha1 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Labels: supportability > Attachments: MAPREDUCE-6751.2.patch, MAPREDUCE-6751.patch > > > There should be a message logged in case of the mapreduce will only spam one > mapper since the source file is compressed with an unsplitable algorithm -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6751) Add debug log message when splitting is not possible because of an unsplittable compression
[ https://issues.apache.org/jira/browse/MAPREDUCE-6751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated MAPREDUCE-6751: -- Summary: Add debug log message when splitting is not possible because of an unsplittable compression (was: Print a log message, when splitting is not possible because of an unsplittable compression) > Add debug log message when splitting is not possible because of an > unsplittable compression > --- > > Key: MAPREDUCE-6751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6751 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client, mrv1, mrv2 >Affects Versions: 3.0.0-alpha1 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Labels: supportability > Attachments: MAPREDUCE-6751.2.patch, MAPREDUCE-6751.patch > > > There should be a message logged in case of the mapreduce will only spam one > mapper since the source file is compressed with an unsplitable algorithm -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6751) Print a log message, when splitting is not possible because of an unsplittable compression
[ https://issues.apache.org/jira/browse/MAPREDUCE-6751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15423234#comment-15423234 ] Ray Chiang commented on MAPREDUCE-6751: --- +1. Looks like a very useful message to me. Committing... > Print a log message, when splitting is not possible because of an > unsplittable compression > -- > > Key: MAPREDUCE-6751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6751 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client, mrv1, mrv2 >Affects Versions: 3.0.0-alpha1 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Labels: supportability > Attachments: MAPREDUCE-6751.2.patch, MAPREDUCE-6751.patch > > > There should be a message logged in case of the mapreduce will only spam one > mapper since the source file is compressed with an unsplitable algorithm -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6751) Print a log message, when splitting is not possible because of an unsplittable compression
[ https://issues.apache.org/jira/browse/MAPREDUCE-6751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated MAPREDUCE-6751: -- Assignee: Peter Vary > Print a log message, when splitting is not possible because of an > unsplittable compression > -- > > Key: MAPREDUCE-6751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6751 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client, mrv1, mrv2 >Affects Versions: 3.0.0-alpha1 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Labels: supportability > Attachments: MAPREDUCE-6751.2.patch, MAPREDUCE-6751.patch > > > There should be a message logged in case of the mapreduce will only spam one > mapper since the source file is compressed with an unsplitable algorithm -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6716) Hadoop shell "mapred job -list" shows improper time format
[
https://issues.apache.org/jira/browse/MAPREDUCE-6716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422755#comment-15422755
]
Jason Lowe commented on MAPREDUCE-6716:
---
The main concern I have here is backward compatibility. The StartTime for
mapred job -list has been that way even back in Hadoop 1.x, and I know people
have written scripts to parse them. This patch would break those scripts, so I
think this patch is appropriate as-is only for Hadoop 3.x. If this needs to go
into 2.x then I think it would be OK to add another option (like '-h' for
human-readability similar to the fs -ls command) to request this formatting.
> Hadoop shell "mapred job -list" shows improper time format
> ---
>
> Key: MAPREDUCE-6716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 2.6.0, 2.8.0
> Environment: centos6.5 x64
> hadoop 2.6.0 /hadoop2.8.0
>Reporter: Shen Yinjie
>Assignee: Shen Yinjie
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6716_trunk.patch, MAPREDUCE_6716.patch
>
>
> when client excute shell "mapred job -list",
> {code} JobId State
> StartTime UserNameQueue Priority
> UsedContainers RsvdContainers
> job_1467197069300_0002 RUNNING 1470405744358
> hdfs default DEFAULT 1
> 0 {code}
> the result shows StartTime is a long type，which is not user-friendly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6757) Multithreaded mapper corrupts buffer pusher in nativetask
[
https://issues.apache.org/jira/browse/MAPREDUCE-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422558#comment-15422558
]
Hadoop QA commented on MAPREDUCE-6757:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 19s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s {color} | {color:green}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask:
The patch generated 0 new + 20 unchanged - 1 fixed = 20 total (was 21) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 51s {color}
| {color:red} hadoop-mapreduce-client-nativetask in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 45s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823862/MAPREDUCE-6757..patch
|
| JIRA Issue | MAPREDUCE-6757 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a4b190d4d453 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ffe1fff |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6672/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6672/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask
|
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6672/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Multithreaded mapp
[jira] [Updated] (MAPREDUCE-6757) Multithreaded mapper corrupts buffer pusher in nativetask
[
https://issues.apache.org/jira/browse/MAPREDUCE-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
He Tianyi updated MAPREDUCE-6757:
-
Status: Patch Available (was: Open)
> Multithreaded mapper corrupts buffer pusher in nativetask
> -
>
> Key: MAPREDUCE-6757
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6757
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nativetask
>Affects Versions: 3.0.0-alpha1
>Reporter: He Tianyi
> Attachments: MAPREDUCE-6757..patch
>
>
> Multiple threads could be calling {{collect}} method of the same
> {{NativeMapOutputCollectorDelegator}} instance at the same time. In this
> case, buffer can be corrupted.
> This may occur when executing Hive queries with custom script.
> Adding 'synchronized' keyword to {{collect}} method would solve the problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6757) Multithreaded mapper corrupts buffer pusher in nativetask
[
https://issues.apache.org/jira/browse/MAPREDUCE-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
He Tianyi updated MAPREDUCE-6757:
-
Attachment: MAPREDUCE-6757..patch
> Multithreaded mapper corrupts buffer pusher in nativetask
> -
>
> Key: MAPREDUCE-6757
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6757
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nativetask
>Affects Versions: 3.0.0-alpha1
>Reporter: He Tianyi
> Attachments: MAPREDUCE-6757..patch
>
>
> Multiple threads could be calling {{collect}} method of the same
> {{NativeMapOutputCollectorDelegator}} instance at the same time. In this
> case, buffer can be corrupted.
> This may occur when executing Hive queries with custom script.
> Adding 'synchronized' keyword to {{collect}} method would solve the problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6757) Multithreaded mapper corrupts buffer pusher in nativetask
He Tianyi created MAPREDUCE-6757:
Summary: Multithreaded mapper corrupts buffer pusher in nativetask
Key: MAPREDUCE-6757
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6757
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: nativetask
Affects Versions: 3.0.0-alpha1
Reporter: He Tianyi
Multiple threads could be calling {{collect}} method of the same
{{NativeMapOutputCollectorDelegator}} instance at the same time. In this case,
buffer can be corrupted.
This may occur when executing Hive queries with custom script.
Adding 'synchronized' keyword to {{collect}} method would solve the problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
