[jira] [Updated] (HADOOP-12258) Need translate java.nio.file.NoSuchFileException to FileNotFoundException in DeprecatedRawLocalFileStatus constructor
[
https://issues.apache.org/jira/browse/HADOOP-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated HADOOP-12258:
---
Description:
need translate java.nio.file.NoSuchFileException to FileNotFoundException in
DeprecatedRawLocalFileStatus constructor.
HADOOP-12045 adds nio to support access time, but nio will create
java.nio.file.NoSuchFileException instead of FileNotFoundException.
many hadoop codes depend on FileNotFoundException to decide whether a file
exists. for example {{FileContext.util().exists()}}.
{code}
public boolean exists(final Path f) throws AccessControlException,
UnsupportedFileSystemException, IOException {
try {
FileStatus fs = FileContext.this.getFileStatus(f);
assert fs != null;
return true;
} catch (FileNotFoundException e) {
return false;
}
}
{code}
same for {{FileSystem#exists}}
{code}
public boolean exists(Path f) throws IOException {
try {
return getFileStatus(f) != null;
} catch (FileNotFoundException e) {
return false;
}
}
{code}
NoSuchFileException will break these functions.
Several test failures for TestDeletionService are caused by this issue:
https://builds.apache.org/job/PreCommit-YARN-Build/8630/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testRelativeDelete/
https://builds.apache.org/job/PreCommit-YARN-Build/8632/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testAbsDelete/
was:
need translate java.nio.file.NoSuchFileException to FileNotFoundException in
DeprecatedRawLocalFileStatus constructor.
HADOOP-12045 adds nio to support access time, but nio will create
java.nio.file.NoSuchFileException instead of FileNotFoundException.
many hadoop codes depend on FileNotFoundException to decide whether a file
exists. for example {{FileContext.util().exists()}}.
{code}
public boolean exists(final Path f) throws AccessControlException,
UnsupportedFileSystemException, IOException {
try {
FileStatus fs = FileContext.this.getFileStatus(f);
assert fs != null;
return true;
} catch (FileNotFoundException e) {
return false;
}
}
{code}
same for {{FileSystem#exists}}
{code}
public boolean exists(Path f) throws IOException {
try {
return getFileStatus(f) != null;
} catch (FileNotFoundException e) {
return false;
}
}
{code}
NoSuchFileException will break these functions.
Several test failures are caused by this issue:
https://builds.apache.org/job/PreCommit-YARN-Build/8630/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testRelativeDelete/
https://builds.apache.org/job/PreCommit-YARN-Build/8632/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testAbsDelete/
> Need translate java.nio.file.NoSuchFileException to FileNotFoundException in
> DeprecatedRawLocalFileStatus constructor
> -
>
> Key: HADOOP-12258
> URL: https://issues.apache.org/jira/browse/HADOOP-12258
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Critical
>
> need translate java.nio.file.NoSuchFileException to FileNotFoundException in
> DeprecatedRawLocalFileStatus constructor.
> HADOOP-12045 adds nio to support access time, but nio will create
> java.nio.file.NoSuchFileException instead of FileNotFoundException.
> many hadoop codes depend on FileNotFoundException to decide whether a file
> exists. for example {{FileContext.util().exists()}}.
> {code}
> public boolean exists(final Path f) throws AccessControlException,
> UnsupportedFileSystemException, IOException {
> try {
> FileStatus fs = FileContext.this.getFileStatus(f);
> assert fs != null;
> return true;
> } catch (FileNotFoundException e) {
> return false;
> }
> }
> {code}
> same for {{FileSystem#exists}}
> {code}
> public boolean exists(Path f) throws IOException {
> try {
> return getFileStatus(f) != null;
> } catch (FileNotFoundException e) {
> return false;
> }
> }
> {code}
> NoSuchFileException will break these functions.
> Several test failures for TestDeletionService are caused by this issue:
> https://builds.apache.org/job/PreCommit-YARN-Build/8630/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testRelativeDelete/
> https://builds.apache.org/job/PreCommit-YARN-Build/8632/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testAbsDelete/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12258) Need translate java.nio.file.NoSuchFileException to FileNotFoundException in DeprecatedRawLocalFileStatus constructor
[
https://issues.apache.org/jira/browse/HADOOP-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated HADOOP-12258:
---
Status: Patch Available (was: Open)
> Need translate java.nio.file.NoSuchFileException to FileNotFoundException in
> DeprecatedRawLocalFileStatus constructor
> -
>
> Key: HADOOP-12258
> URL: https://issues.apache.org/jira/browse/HADOOP-12258
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Critical
> Attachments: HADOOP-12258.000.patch
>
>
> need translate java.nio.file.NoSuchFileException to FileNotFoundException in
> DeprecatedRawLocalFileStatus constructor.
> HADOOP-12045 adds nio to support access time, but nio will create
> java.nio.file.NoSuchFileException instead of FileNotFoundException.
> many hadoop codes depend on FileNotFoundException to decide whether a file
> exists. for example {{FileContext.util().exists()}}.
> {code}
> public boolean exists(final Path f) throws AccessControlException,
> UnsupportedFileSystemException, IOException {
> try {
> FileStatus fs = FileContext.this.getFileStatus(f);
> assert fs != null;
> return true;
> } catch (FileNotFoundException e) {
> return false;
> }
> }
> {code}
> same for {{FileSystem#exists}}
> {code}
> public boolean exists(Path f) throws IOException {
> try {
> return getFileStatus(f) != null;
> } catch (FileNotFoundException e) {
> return false;
> }
> }
> {code}
> NoSuchFileException will break these functions.
> Several test failures for TestDeletionService are caused by this issue:
> https://builds.apache.org/job/PreCommit-YARN-Build/8630/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testRelativeDelete/
> https://builds.apache.org/job/PreCommit-YARN-Build/8632/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testAbsDelete/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12258) Need translate java.nio.file.NoSuchFileException to FileNotFoundException in DeprecatedRawLocalFileStatus constructor
[
https://issues.apache.org/jira/browse/HADOOP-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated HADOOP-12258:
---
Attachment: HADOOP-12258.000.patch
> Need translate java.nio.file.NoSuchFileException to FileNotFoundException in
> DeprecatedRawLocalFileStatus constructor
> -
>
> Key: HADOOP-12258
> URL: https://issues.apache.org/jira/browse/HADOOP-12258
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Critical
> Attachments: HADOOP-12258.000.patch
>
>
> need translate java.nio.file.NoSuchFileException to FileNotFoundException in
> DeprecatedRawLocalFileStatus constructor.
> HADOOP-12045 adds nio to support access time, but nio will create
> java.nio.file.NoSuchFileException instead of FileNotFoundException.
> many hadoop codes depend on FileNotFoundException to decide whether a file
> exists. for example {{FileContext.util().exists()}}.
> {code}
> public boolean exists(final Path f) throws AccessControlException,
> UnsupportedFileSystemException, IOException {
> try {
> FileStatus fs = FileContext.this.getFileStatus(f);
> assert fs != null;
> return true;
> } catch (FileNotFoundException e) {
> return false;
> }
> }
> {code}
> same for {{FileSystem#exists}}
> {code}
> public boolean exists(Path f) throws IOException {
> try {
> return getFileStatus(f) != null;
> } catch (FileNotFoundException e) {
> return false;
> }
> }
> {code}
> NoSuchFileException will break these functions.
> Several test failures for TestDeletionService are caused by this issue:
> https://builds.apache.org/job/PreCommit-YARN-Build/8630/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testRelativeDelete/
> https://builds.apache.org/job/PreCommit-YARN-Build/8632/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testAbsDelete/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12258) Need translate java.nio.file.NoSuchFileException to FileNotFoundException in DeprecatedRawLocalFileStatus constructor
[
https://issues.apache.org/jira/browse/HADOOP-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated HADOOP-12258:
---
Description:
need translate java.nio.file.NoSuchFileException to FileNotFoundException in
DeprecatedRawLocalFileStatus constructor.
HADOOP-12045 adds nio to support access time, but nio will create
java.nio.file.NoSuchFileException instead of FileNotFoundException.
many hadoop codes depend on FileNotFoundException to decide whether a file
exists. for example {{FileContext.util().exists()}}.
{code}
public boolean exists(final Path f) throws AccessControlException,
UnsupportedFileSystemException, IOException {
try {
FileStatus fs = FileContext.this.getFileStatus(f);
assert fs != null;
return true;
} catch (FileNotFoundException e) {
return false;
}
}
{code}
same for {{FileSystem#exists}}
{code}
public boolean exists(Path f) throws IOException {
try {
return getFileStatus(f) != null;
} catch (FileNotFoundException e) {
return false;
}
}
{code}
NoSuchFileException will break these functions.
Since {{exists}} is one of the most used API in FileSystem, this issue is very
critical.
Several test failures for TestDeletionService are caused by this issue:
https://builds.apache.org/job/PreCommit-YARN-Build/8630/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testRelativeDelete/
https://builds.apache.org/job/PreCommit-YARN-Build/8632/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testAbsDelete/
was:
need translate java.nio.file.NoSuchFileException to FileNotFoundException in
DeprecatedRawLocalFileStatus constructor.
HADOOP-12045 adds nio to support access time, but nio will create
java.nio.file.NoSuchFileException instead of FileNotFoundException.
many hadoop codes depend on FileNotFoundException to decide whether a file
exists. for example {{FileContext.util().exists()}}.
{code}
public boolean exists(final Path f) throws AccessControlException,
UnsupportedFileSystemException, IOException {
try {
FileStatus fs = FileContext.this.getFileStatus(f);
assert fs != null;
return true;
} catch (FileNotFoundException e) {
return false;
}
}
{code}
same for {{FileSystem#exists}}
{code}
public boolean exists(Path f) throws IOException {
try {
return getFileStatus(f) != null;
} catch (FileNotFoundException e) {
return false;
}
}
{code}
NoSuchFileException will break these functions.
Several test failures for TestDeletionService are caused by this issue:
https://builds.apache.org/job/PreCommit-YARN-Build/8630/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testRelativeDelete/
https://builds.apache.org/job/PreCommit-YARN-Build/8632/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testAbsDelete/
> Need translate java.nio.file.NoSuchFileException to FileNotFoundException in
> DeprecatedRawLocalFileStatus constructor
> -
>
> Key: HADOOP-12258
> URL: https://issues.apache.org/jira/browse/HADOOP-12258
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Critical
> Attachments: HADOOP-12258.000.patch
>
>
> need translate java.nio.file.NoSuchFileException to FileNotFoundException in
> DeprecatedRawLocalFileStatus constructor.
> HADOOP-12045 adds nio to support access time, but nio will create
> java.nio.file.NoSuchFileException instead of FileNotFoundException.
> many hadoop codes depend on FileNotFoundException to decide whether a file
> exists. for example {{FileContext.util().exists()}}.
> {code}
> public boolean exists(final Path f) throws AccessControlException,
> UnsupportedFileSystemException, IOException {
> try {
> FileStatus fs = FileContext.this.getFileStatus(f);
> assert fs != null;
> return true;
> } catch (FileNotFoundException e) {
> return false;
> }
> }
> {code}
> same for {{FileSystem#exists}}
> {code}
> public boolean exists(Path f) throws IOException {
> try {
> return getFileStatus(f) != null;
> } catch (FileNotFoundException e) {
> return false;
> }
> }
> {code}
> NoSuchFileException will break these functions.
> Since {{exists}} is one of the most used API in FileSystem, this issue is
> very critical.
> Several test failures for TestDeletionService are caused by this issue:
> https://builds.apache.org/job/PreCommit-YARN-Build/8630/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testRelativeDelete/
> https://builds.apache.org/jo
[jira] [Commented] (HADOOP-12258) Need translate java.nio.file.NoSuchFileException to FileNotFoundException in DeprecatedRawLocalFileStatus constructor
[
https://issues.apache.org/jira/browse/HADOOP-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638363#comment-14638363
]
zhihai xu commented on HADOOP-12258:
I uploaded a patch HADOOP-12258.000.patch for review. The patch will throw
FileNotFoundException instead of NoSuchFileException to make it backward
compatible. We may also change FileSystem#exists and
FileContext.util().exists() to fix this issue, but there are many old code
which may still catch FileNotFoundException instead NoSuchFileException when
call getFileStatus.
For example:DirectoryCollection#createDir. It will be good to make it
compatible with old code.
> Need translate java.nio.file.NoSuchFileException to FileNotFoundException in
> DeprecatedRawLocalFileStatus constructor
> -
>
> Key: HADOOP-12258
> URL: https://issues.apache.org/jira/browse/HADOOP-12258
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Critical
> Attachments: HADOOP-12258.000.patch
>
>
> need translate java.nio.file.NoSuchFileException to FileNotFoundException in
> DeprecatedRawLocalFileStatus constructor.
> HADOOP-12045 adds nio to support access time, but nio will create
> java.nio.file.NoSuchFileException instead of FileNotFoundException.
> many hadoop codes depend on FileNotFoundException to decide whether a file
> exists. for example {{FileContext.util().exists()}}.
> {code}
> public boolean exists(final Path f) throws AccessControlException,
> UnsupportedFileSystemException, IOException {
> try {
> FileStatus fs = FileContext.this.getFileStatus(f);
> assert fs != null;
> return true;
> } catch (FileNotFoundException e) {
> return false;
> }
> }
> {code}
> same for {{FileSystem#exists}}
> {code}
> public boolean exists(Path f) throws IOException {
> try {
> return getFileStatus(f) != null;
> } catch (FileNotFoundException e) {
> return false;
> }
> }
> {code}
> NoSuchFileException will break these functions.
> Since {{exists}} is one of the most used API in FileSystem, this issue is
> very critical.
> Several test failures for TestDeletionService are caused by this issue:
> https://builds.apache.org/job/PreCommit-YARN-Build/8630/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testRelativeDelete/
> https://builds.apache.org/job/PreCommit-YARN-Build/8632/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testAbsDelete/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12258) Need translate java.nio.file.NoSuchFileException to FileNotFoundException in DeprecatedRawLocalFileStatus constructor
[
https://issues.apache.org/jira/browse/HADOOP-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638459#comment-14638459
]
Hadoop QA commented on HADOOP-12258:
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 58s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:red}-1{color} | javac | 8m 9s | The applied patch generated 1
additional warning messages. |
| {color:green}+1{color} | javadoc | 11m 15s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 32s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 20s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 55s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 40s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 35s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 26m 13s | Tests passed in
hadoop-common. |
| | | 70m 41s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12746735/HADOOP-12258.000.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ee98d63 |
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7329/artifact/patchprocess/diffJavacWarnings.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7329/artifact/patchprocess/testrun_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7329/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7329/console |
This message was automatically generated.
> Need translate java.nio.file.NoSuchFileException to FileNotFoundException in
> DeprecatedRawLocalFileStatus constructor
> -
>
> Key: HADOOP-12258
> URL: https://issues.apache.org/jira/browse/HADOOP-12258
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Critical
> Attachments: HADOOP-12258.000.patch
>
>
> need translate java.nio.file.NoSuchFileException to FileNotFoundException in
> DeprecatedRawLocalFileStatus constructor.
> HADOOP-12045 adds nio to support access time, but nio will create
> java.nio.file.NoSuchFileException instead of FileNotFoundException.
> many hadoop codes depend on FileNotFoundException to decide whether a file
> exists. for example {{FileContext.util().exists()}}.
> {code}
> public boolean exists(final Path f) throws AccessControlException,
> UnsupportedFileSystemException, IOException {
> try {
> FileStatus fs = FileContext.this.getFileStatus(f);
> assert fs != null;
> return true;
> } catch (FileNotFoundException e) {
> return false;
> }
> }
> {code}
> same for {{FileSystem#exists}}
> {code}
> public boolean exists(Path f) throws IOException {
> try {
> return getFileStatus(f) != null;
> } catch (FileNotFoundException e) {
> return false;
> }
> }
> {code}
> NoSuchFileException will break these functions.
> Since {{exists}} is one of the most used API in FileSystem, this issue is
> very critical.
> Several test failures for TestDeletionService are caused by this issue:
> https://builds.apache.org/job/PreCommit-YARN-Build/8630/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testRelativeDelete/
> https://builds.apache.org/job/PreCommit-YARN-Build/8632/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testAbsDelete/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12260) BlockSender.sendChunks() exception
Marius created HADOOP-12260: --- Summary: BlockSender.sendChunks() exception Key: HADOOP-12260 URL: https://issues.apache.org/jira/browse/HADOOP-12260 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.7.1, 2.6.0 Environment: OS: CentOS Linux release 7.1.1503 (Core) Kernel: 3.10.0-229.1.2.el7.x86_64 Reporter: Marius I was running some streaming jobs with avro files from my hadoop cluster. They performed poorly so i checked the logs of my datanodes and found this: http://pastebin.com/DXKJJ55z The cluster is running on CentOS machines: CentOS Linux release 7.1.1503 (Core) This is the Kernel: 3.10.0-229.1.2.el7.x86_64 No one one the userlist replied and i could not find anything helpful on the internet despite disk failure which is unlikely to cause this because here are several machines and its not very likely that all of their disks fail at the same time. This error is not reported on the console when running a job and the error occurs from time to time and then dissapears and comes back again. The block size of the cluster is the default value. This is my command: hadoop jar hadoop-streaming-2.7.1.jar -files mapper.py,reducer.py,avro-1. 7.7.jar,avro-mapred-1.7.7-hadoop2.jar -D mapreduce.job.reduces=15 -libjars avro-1.7.7.jar,avro-mapred-1.7.7-hadoop2.jar -input /Y/Y1.avro -output /htest/output -mapper mapper.py -reducer reducer.py -inputformat org.apache.avro.mapred.AvroAsTextInputFormat Marius -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12260) BlockSender.sendChunks() exception
[ https://issues.apache.org/jira/browse/HADOOP-12260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marius updated HADOOP-12260: Description: Hi I was running some streaming jobs with avro files from my hadoop cluster. They performed poorly so i checked the logs of my datanodes and found this: http://pastebin.com/DXKJJ55z The cluster is running on CentOS machines: CentOS Linux release 7.1.1503 (Core) This is the Kernel: 3.10.0-229.1.2.el7.x86_64 No one one the userlist replied and i could not find anything helpful on the internet despite disk failure which is unlikely to cause this because here are several machines and its not very likely that all of their disks fail at the same time. This error is not reported on the console when running a job and the error occurs from time to time and then dissapears and comes back again. The block size of the cluster is the default value. This is my command: hadoop jar hadoop-streaming-2.7.1.jar -files mapper.py,reducer.py,avro-1. 7.7.jar,avro-mapred-1.7.7-hadoop2.jar -D mapreduce.job.reduces=15 -libjars avro-1.7.7.jar,avro-mapred-1.7.7-hadoop2.jar -input /Y/Y1.avro -output /htest/output -mapper mapper.py -reducer reducer.py -inputformat org.apache.avro.mapred.AvroAsTextInputFormat Marius was: I was running some streaming jobs with avro files from my hadoop cluster. They performed poorly so i checked the logs of my datanodes and found this: http://pastebin.com/DXKJJ55z The cluster is running on CentOS machines: CentOS Linux release 7.1.1503 (Core) This is the Kernel: 3.10.0-229.1.2.el7.x86_64 No one one the userlist replied and i could not find anything helpful on the internet despite disk failure which is unlikely to cause this because here are several machines and its not very likely that all of their disks fail at the same time. This error is not reported on the console when running a job and the error occurs from time to time and then dissapears and comes back again. The block size of the cluster is the default value. This is my command: hadoop jar hadoop-streaming-2.7.1.jar -files mapper.py,reducer.py,avro-1. 7.7.jar,avro-mapred-1.7.7-hadoop2.jar -D mapreduce.job.reduces=15 -libjars avro-1.7.7.jar,avro-mapred-1.7.7-hadoop2.jar -input /Y/Y1.avro -output /htest/output -mapper mapper.py -reducer reducer.py -inputformat org.apache.avro.mapred.AvroAsTextInputFormat Marius > BlockSender.sendChunks() exception > -- > > Key: HADOOP-12260 > URL: https://issues.apache.org/jira/browse/HADOOP-12260 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.0, 2.7.1 > Environment: OS: CentOS Linux release 7.1.1503 (Core) > Kernel: 3.10.0-229.1.2.el7.x86_64 >Reporter: Marius > > Hi > I was running some streaming jobs with avro files from my hadoop cluster. > They performed poorly so i checked the logs of my datanodes and found this: > http://pastebin.com/DXKJJ55z > The cluster is running on CentOS machines: > CentOS Linux release 7.1.1503 (Core) > This is the Kernel: > 3.10.0-229.1.2.el7.x86_64 > No one one the userlist replied and i could not find anything helpful on the > internet despite disk failure which is unlikely to cause this because here > are several machines and its not very likely that all of their disks fail at > the same time. > This error is not reported on the console when running a job and the error > occurs from time to time and then dissapears and comes back again. > The block size of the cluster is the default value. > This is my command: > hadoop jar hadoop-streaming-2.7.1.jar -files mapper.py,reducer.py,avro-1. > 7.7.jar,avro-mapred-1.7.7-hadoop2.jar -D mapreduce.job.reduces=15 -libjars > avro-1.7.7.jar,avro-mapred-1.7.7-hadoop2.jar -input /Y/Y1.avro -output > /htest/output -mapper mapper.py -reducer reducer.py -inputformat > org.apache.avro.mapred.AvroAsTextInputFormat > Marius -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12260) BlockSender.sendChunks() exception
[ https://issues.apache.org/jira/browse/HADOOP-12260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marius updated HADOOP-12260: Description: Hi I was running some streaming jobs with avro files from my hadoop cluster. They performed poorly so i checked the logs of my datanodes and found this: http://pastebin.com/DXKJJ55z The cluster is running on CentOS machines: CentOS Linux release 7.1.1503 (Core) This is the Kernel: 3.10.0-229.1.2.el7.x86_64 No one on the userlist replied and i could not find anything helpful on the internet despite disk failure which is unlikely to cause this because here are several machines and its not very likely that all of their disks fail at the same time. This error is not reported on the console when running a job and the error occurs from time to time and then dissapears and comes back again. The block size of the cluster is the default value. This is my command: hadoop jar hadoop-streaming-2.7.1.jar -files mapper.py,reducer.py,avro-1. 7.7.jar,avro-mapred-1.7.7-hadoop2.jar -D mapreduce.job.reduces=15 -libjars avro-1.7.7.jar,avro-mapred-1.7.7-hadoop2.jar -input /Y/Y1.avro -output /htest/output -mapper mapper.py -reducer reducer.py -inputformat org.apache.avro.mapred.AvroAsTextInputFormat Marius was: Hi I was running some streaming jobs with avro files from my hadoop cluster. They performed poorly so i checked the logs of my datanodes and found this: http://pastebin.com/DXKJJ55z The cluster is running on CentOS machines: CentOS Linux release 7.1.1503 (Core) This is the Kernel: 3.10.0-229.1.2.el7.x86_64 No one one the userlist replied and i could not find anything helpful on the internet despite disk failure which is unlikely to cause this because here are several machines and its not very likely that all of their disks fail at the same time. This error is not reported on the console when running a job and the error occurs from time to time and then dissapears and comes back again. The block size of the cluster is the default value. This is my command: hadoop jar hadoop-streaming-2.7.1.jar -files mapper.py,reducer.py,avro-1. 7.7.jar,avro-mapred-1.7.7-hadoop2.jar -D mapreduce.job.reduces=15 -libjars avro-1.7.7.jar,avro-mapred-1.7.7-hadoop2.jar -input /Y/Y1.avro -output /htest/output -mapper mapper.py -reducer reducer.py -inputformat org.apache.avro.mapred.AvroAsTextInputFormat Marius > BlockSender.sendChunks() exception > -- > > Key: HADOOP-12260 > URL: https://issues.apache.org/jira/browse/HADOOP-12260 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.0, 2.7.1 > Environment: OS: CentOS Linux release 7.1.1503 (Core) > Kernel: 3.10.0-229.1.2.el7.x86_64 >Reporter: Marius > > Hi > I was running some streaming jobs with avro files from my hadoop cluster. > They performed poorly so i checked the logs of my datanodes and found this: > http://pastebin.com/DXKJJ55z > The cluster is running on CentOS machines: > CentOS Linux release 7.1.1503 (Core) > This is the Kernel: > 3.10.0-229.1.2.el7.x86_64 > No one on the userlist replied and i could not find anything helpful on the > internet despite disk failure which is unlikely to cause this because here > are several machines and its not very likely that all of their disks fail at > the same time. > This error is not reported on the console when running a job and the error > occurs from time to time and then dissapears and comes back again. > The block size of the cluster is the default value. > This is my command: > hadoop jar hadoop-streaming-2.7.1.jar -files mapper.py,reducer.py,avro-1. > 7.7.jar,avro-mapred-1.7.7-hadoop2.jar -D mapreduce.job.reduces=15 -libjars > avro-1.7.7.jar,avro-mapred-1.7.7-hadoop2.jar -input /Y/Y1.avro -output > /htest/output -mapper mapper.py -reducer reducer.py -inputformat > org.apache.avro.mapred.AvroAsTextInputFormat > Marius -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12184) Remove unused Linux-specific constants in NativeIO
[ https://issues.apache.org/jira/browse/HADOOP-12184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638675#comment-14638675 ] Hudson commented on HADOOP-12184: - FAILURE: Integrated in Hadoop-Yarn-trunk #995 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/995/]) HADOOP-12184. Remove unused Linux-specific constants in NativeIO (Martin Walsh via Colin P. McCabe) (cmccabe: rev 1b3bceb58c8e536a75fa3f99cc3ceeaba91a07de) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java * hadoop-common-project/hadoop-common/CHANGES.txt > Remove unused Linux-specific constants in NativeIO > -- > > Key: HADOOP-12184 > URL: https://issues.apache.org/jira/browse/HADOOP-12184 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: 2.7.0 > Environment: Solaris >Reporter: Martin Walsh >Assignee: Martin Walsh > Fix For: 2.8.0 > > Attachments: HADOOP-12184.001.patch > > > The POSIX subclass in NativeIO contains the following constants that are not > defined in POSIX: > O_ASYNC, O_FSYNC, O_NDELAY > Stat.S_IFWHT > These constants are not used within Hadoop, as far as I could tell. As they > are not part of the POSIX specification, they should be removed as part of an > effort to create portable code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12239) StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB
[
https://issues.apache.org/jira/browse/HADOOP-12239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638676#comment-14638676
]
Hudson commented on HADOOP-12239:
-
FAILURE: Integrated in Hadoop-Yarn-trunk #995 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/995/])
HADOOP-12239. StorageException complaining " no lease ID" when updating
FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev
efa97243ecb84b3b468e732897cd685e3869f480)
* hadoop-common-project/hadoop-common/CHANGES.txt
*
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
> StorageException complaining " no lease ID" when updating
> FolderLastModifiedTime in WASB
>
>
> Key: HADOOP-12239
> URL: https://issues.apache.org/jira/browse/HADOOP-12239
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, tools
>Affects Versions: 2.7.0
> Environment: Windows Azure
>Reporter: Duo Xu
>Assignee: Duo Xu
> Fix For: 2.8.0
>
> Attachments: HADOOP-12239.01.patch
>
>
> This is a similar issue as HADOOP-11523 and HADOOP-12089, which I found in a
> customer's HBase cluster logs, but the piece of code is in a different place.
> {code}
> 2015-07-09 13:38:57,388 INFO org.apache.hadoop.hbase.master.SplitLogManager:
> dead splitlog workers
> [workernode3.xxx.b6.internal.cloudapp.net,60020,1436448555180]
> 2015-07-09 13:38:57,466 ERROR org.apache.hadoop.hbase.executor.EventHandler:
> Caught throwable while processing event M_SERVER_SHUTDOWN
> java.io.IOException: failed log splitting for
> workernode12.xxx.b6.internal.cloudapp.net,60020,1436 448566374, will retry
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:343)
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:211)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unable to write RenamePending file for folder
> rename from
> hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374 to
> hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374-splitting
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:258)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.prepareAtomicFolderRename(NativeAzureFileSystem.java:2110)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1998)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.getLogDirs(MasterFileSystem.java:325)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:412)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:390)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:288)
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:204)
> ... 4 more
> Caused by: org.apache.hadoop.fs.azure.AzureException:
> com.microsoft.azure.storage.StorageException: There is currently a lease on
> the blob and no lease ID was specified in the request.
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2598)
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2609)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1366)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1195)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:775)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:255)
> ... 11 more
> Caused by: com.microsoft.azure.storage.StorageException: There is currently a
> lease on the blob and no lease ID was specified in the request.
> at
> com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
> at
> com.microsoft.azure.storage.core.StorageReque
[jira] [Commented] (HADOOP-12184) Remove unused Linux-specific constants in NativeIO
[ https://issues.apache.org/jira/browse/HADOOP-12184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638697#comment-14638697 ] Hudson commented on HADOOP-12184: - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #265 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/265/]) HADOOP-12184. Remove unused Linux-specific constants in NativeIO (Martin Walsh via Colin P. McCabe) (cmccabe: rev 1b3bceb58c8e536a75fa3f99cc3ceeaba91a07de) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java * hadoop-common-project/hadoop-common/CHANGES.txt > Remove unused Linux-specific constants in NativeIO > -- > > Key: HADOOP-12184 > URL: https://issues.apache.org/jira/browse/HADOOP-12184 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: 2.7.0 > Environment: Solaris >Reporter: Martin Walsh >Assignee: Martin Walsh > Fix For: 2.8.0 > > Attachments: HADOOP-12184.001.patch > > > The POSIX subclass in NativeIO contains the following constants that are not > defined in POSIX: > O_ASYNC, O_FSYNC, O_NDELAY > Stat.S_IFWHT > These constants are not used within Hadoop, as far as I could tell. As they > are not part of the POSIX specification, they should be removed as part of an > effort to create portable code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12239) StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB
[
https://issues.apache.org/jira/browse/HADOOP-12239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638698#comment-14638698
]
Hudson commented on HADOOP-12239:
-
SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #265 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/265/])
HADOOP-12239. StorageException complaining " no lease ID" when updating
FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev
efa97243ecb84b3b468e732897cd685e3869f480)
*
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
* hadoop-common-project/hadoop-common/CHANGES.txt
> StorageException complaining " no lease ID" when updating
> FolderLastModifiedTime in WASB
>
>
> Key: HADOOP-12239
> URL: https://issues.apache.org/jira/browse/HADOOP-12239
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, tools
>Affects Versions: 2.7.0
> Environment: Windows Azure
>Reporter: Duo Xu
>Assignee: Duo Xu
> Fix For: 2.8.0
>
> Attachments: HADOOP-12239.01.patch
>
>
> This is a similar issue as HADOOP-11523 and HADOOP-12089, which I found in a
> customer's HBase cluster logs, but the piece of code is in a different place.
> {code}
> 2015-07-09 13:38:57,388 INFO org.apache.hadoop.hbase.master.SplitLogManager:
> dead splitlog workers
> [workernode3.xxx.b6.internal.cloudapp.net,60020,1436448555180]
> 2015-07-09 13:38:57,466 ERROR org.apache.hadoop.hbase.executor.EventHandler:
> Caught throwable while processing event M_SERVER_SHUTDOWN
> java.io.IOException: failed log splitting for
> workernode12.xxx.b6.internal.cloudapp.net,60020,1436 448566374, will retry
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:343)
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:211)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unable to write RenamePending file for folder
> rename from
> hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374 to
> hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374-splitting
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:258)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.prepareAtomicFolderRename(NativeAzureFileSystem.java:2110)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1998)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.getLogDirs(MasterFileSystem.java:325)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:412)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:390)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:288)
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:204)
> ... 4 more
> Caused by: org.apache.hadoop.fs.azure.AzureException:
> com.microsoft.azure.storage.StorageException: There is currently a lease on
> the blob and no lease ID was specified in the request.
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2598)
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2609)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1366)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1195)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:775)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:255)
> ... 11 more
> Caused by: com.microsoft.azure.storage.StorageException: There is currently a
> lease on the blob and no lease ID was specified in the request.
> at
> com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
> at
> com.microsoft.azure.storage.core.
[jira] [Updated] (HADOOP-12170) hadoop-common's JNIFlags.cmake is redundant and can be removed
[ https://issues.apache.org/jira/browse/HADOOP-12170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Burlison updated HADOOP-12170: --- Attachment: (was: HADOOP-12170) > hadoop-common's JNIFlags.cmake is redundant and can be removed > -- > > Key: HADOOP-12170 > URL: https://issues.apache.org/jira/browse/HADOOP-12170 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: 2.7.1 >Reporter: Alan Burlison >Assignee: Alan Burlison >Priority: Minor > Attachments: HADOOP-12170.001.patch > > > With the integration of: > * HADOOP-12036 Consolidate all of the cmake extensions in one *directory > * HADOOP-12104 Migrate Hadoop Pipes native build to new CMake > * HDFS-8635 Migrate HDFS native build to new CMake framework > * MAPREDUCE-6407 Migrate MAPREDUCE native build to new CMake > * YARN-3827 Migrate YARN native build to new CMake framework > hadoop-common-project/hadoop-common/src/JNIFlags.cmake is now redundant and > can be removed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12170) hadoop-common's JNIFlags.cmake is redundant and can be removed
[ https://issues.apache.org/jira/browse/HADOOP-12170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Burlison updated HADOOP-12170: --- Attachment: HADOOP-12170.001.patch > hadoop-common's JNIFlags.cmake is redundant and can be removed > -- > > Key: HADOOP-12170 > URL: https://issues.apache.org/jira/browse/HADOOP-12170 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: 2.7.1 >Reporter: Alan Burlison >Assignee: Alan Burlison >Priority: Minor > Attachments: HADOOP-12170.001.patch > > > With the integration of: > * HADOOP-12036 Consolidate all of the cmake extensions in one *directory > * HADOOP-12104 Migrate Hadoop Pipes native build to new CMake > * HDFS-8635 Migrate HDFS native build to new CMake framework > * MAPREDUCE-6407 Migrate MAPREDUCE native build to new CMake > * YARN-3827 Migrate YARN native build to new CMake framework > hadoop-common-project/hadoop-common/src/JNIFlags.cmake is now redundant and > can be removed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11505) hadoop-mapreduce-client-nativetask uses bswap where be32toh is needed, doesn't work on non-x86
[ https://issues.apache.org/jira/browse/HADOOP-11505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Burlison updated HADOOP-11505: --- Attachment: (was: HADOOP-11505) > hadoop-mapreduce-client-nativetask uses bswap where be32toh is needed, > doesn't work on non-x86 > -- > > Key: HADOOP-11505 > URL: https://issues.apache.org/jira/browse/HADOOP-11505 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Labels: BB2015-05-TBR > Attachments: HADOOP-11505.001.patch, HADOOP-11505.003.patch > > > hadoop-mapreduce-client-nativetask fails to use x86 optimizations in some > cases. Also, on some alternate, non-x86, non-ARM architectures the generated > code is incorrect. Thanks to Steve Loughran and Edward Nevill for finding > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12239) StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB
[
https://issues.apache.org/jira/browse/HADOOP-12239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638784#comment-14638784
]
Hudson commented on HADOOP-12239:
-
FAILURE: Integrated in Hadoop-Hdfs-trunk #2192 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2192/])
HADOOP-12239. StorageException complaining " no lease ID" when updating
FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev
efa97243ecb84b3b468e732897cd685e3869f480)
*
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
* hadoop-common-project/hadoop-common/CHANGES.txt
> StorageException complaining " no lease ID" when updating
> FolderLastModifiedTime in WASB
>
>
> Key: HADOOP-12239
> URL: https://issues.apache.org/jira/browse/HADOOP-12239
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, tools
>Affects Versions: 2.7.0
> Environment: Windows Azure
>Reporter: Duo Xu
>Assignee: Duo Xu
> Fix For: 2.8.0
>
> Attachments: HADOOP-12239.01.patch
>
>
> This is a similar issue as HADOOP-11523 and HADOOP-12089, which I found in a
> customer's HBase cluster logs, but the piece of code is in a different place.
> {code}
> 2015-07-09 13:38:57,388 INFO org.apache.hadoop.hbase.master.SplitLogManager:
> dead splitlog workers
> [workernode3.xxx.b6.internal.cloudapp.net,60020,1436448555180]
> 2015-07-09 13:38:57,466 ERROR org.apache.hadoop.hbase.executor.EventHandler:
> Caught throwable while processing event M_SERVER_SHUTDOWN
> java.io.IOException: failed log splitting for
> workernode12.xxx.b6.internal.cloudapp.net,60020,1436 448566374, will retry
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:343)
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:211)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unable to write RenamePending file for folder
> rename from
> hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374 to
> hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374-splitting
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:258)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.prepareAtomicFolderRename(NativeAzureFileSystem.java:2110)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1998)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.getLogDirs(MasterFileSystem.java:325)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:412)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:390)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:288)
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:204)
> ... 4 more
> Caused by: org.apache.hadoop.fs.azure.AzureException:
> com.microsoft.azure.storage.StorageException: There is currently a lease on
> the blob and no lease ID was specified in the request.
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2598)
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2609)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1366)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1195)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:775)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:255)
> ... 11 more
> Caused by: com.microsoft.azure.storage.StorageException: There is currently a
> lease on the blob and no lease ID was specified in the request.
> at
> com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
> at
> com.microsoft.azure.storage.core.StorageReq
[jira] [Commented] (HADOOP-12184) Remove unused Linux-specific constants in NativeIO
[ https://issues.apache.org/jira/browse/HADOOP-12184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638783#comment-14638783 ] Hudson commented on HADOOP-12184: - FAILURE: Integrated in Hadoop-Hdfs-trunk #2192 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2192/]) HADOOP-12184. Remove unused Linux-specific constants in NativeIO (Martin Walsh via Colin P. McCabe) (cmccabe: rev 1b3bceb58c8e536a75fa3f99cc3ceeaba91a07de) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java * hadoop-common-project/hadoop-common/CHANGES.txt > Remove unused Linux-specific constants in NativeIO > -- > > Key: HADOOP-12184 > URL: https://issues.apache.org/jira/browse/HADOOP-12184 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: 2.7.0 > Environment: Solaris >Reporter: Martin Walsh >Assignee: Martin Walsh > Fix For: 2.8.0 > > Attachments: HADOOP-12184.001.patch > > > The POSIX subclass in NativeIO contains the following constants that are not > defined in POSIX: > O_ASYNC, O_FSYNC, O_NDELAY > Stat.S_IFWHT > These constants are not used within Hadoop, as far as I could tell. As they > are not part of the POSIX specification, they should be removed as part of an > effort to create portable code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12170) hadoop-common's JNIFlags.cmake is redundant and can be removed
[
https://issues.apache.org/jira/browse/HADOOP-12170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638793#comment-14638793
]
Hadoop QA commented on HADOOP-12170:
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 6m 9s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 8m 27s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 19s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 25s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | common tests | 22m 26s | Tests failed in
hadoop-common. |
| | | 39m 24s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.fs.TestLocalFsFCStatistics |
| | hadoop.fs.sftp.TestSFTPFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12746791/HADOOP-12170.001.patch
|
| Optional Tests | javac unit |
| git revision | trunk / ee98d63 |
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7330/artifact/patchprocess/testrun_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7330/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7330/console |
This message was automatically generated.
> hadoop-common's JNIFlags.cmake is redundant and can be removed
> --
>
> Key: HADOOP-12170
> URL: https://issues.apache.org/jira/browse/HADOOP-12170
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
>Affects Versions: 2.7.1
>Reporter: Alan Burlison
>Assignee: Alan Burlison
>Priority: Minor
> Attachments: HADOOP-12170.001.patch
>
>
> With the integration of:
> * HADOOP-12036 Consolidate all of the cmake extensions in one *directory
> * HADOOP-12104 Migrate Hadoop Pipes native build to new CMake
> * HDFS-8635 Migrate HDFS native build to new CMake framework
> * MAPREDUCE-6407 Migrate MAPREDUCE native build to new CMake
> * YARN-3827 Migrate YARN native build to new CMake framework
> hadoop-common-project/hadoop-common/src/JNIFlags.cmake is now redundant and
> can be removed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12260) BlockSender.sendChunks() exception
[ https://issues.apache.org/jira/browse/HADOOP-12260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638820#comment-14638820 ] Marius commented on HADOOP-12260: - I was able to recreate this error on fresh machine with a single node setup running Linux Mint 17.2. The error occured in the same was as before even with a different avro file (took this: https://cwiki.apache.org/confluence/display/AVRO/Interoperability+Testing but wrote a 100MB file by looping over append DATUM). The error occured with pydoop as well as with the normal streaming command. Marius > BlockSender.sendChunks() exception > -- > > Key: HADOOP-12260 > URL: https://issues.apache.org/jira/browse/HADOOP-12260 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.0, 2.7.1 > Environment: OS: CentOS Linux release 7.1.1503 (Core) > Kernel: 3.10.0-229.1.2.el7.x86_64 >Reporter: Marius > > Hi > I was running some streaming jobs with avro files from my hadoop cluster. > They performed poorly so i checked the logs of my datanodes and found this: > http://pastebin.com/DXKJJ55z > The cluster is running on CentOS machines: > CentOS Linux release 7.1.1503 (Core) > This is the Kernel: > 3.10.0-229.1.2.el7.x86_64 > No one on the userlist replied and i could not find anything helpful on the > internet despite disk failure which is unlikely to cause this because here > are several machines and its not very likely that all of their disks fail at > the same time. > This error is not reported on the console when running a job and the error > occurs from time to time and then dissapears and comes back again. > The block size of the cluster is the default value. > This is my command: > hadoop jar hadoop-streaming-2.7.1.jar -files mapper.py,reducer.py,avro-1. > 7.7.jar,avro-mapred-1.7.7-hadoop2.jar -D mapreduce.job.reduces=15 -libjars > avro-1.7.7.jar,avro-mapred-1.7.7-hadoop2.jar -input /Y/Y1.avro -output > /htest/output -mapper mapper.py -reducer reducer.py -inputformat > org.apache.avro.mapred.AvroAsTextInputFormat > Marius -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12260) BlockSender.sendChunks() exception
[ https://issues.apache.org/jira/browse/HADOOP-12260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marius updated HADOOP-12260: Tags: avro, sendChunks error > BlockSender.sendChunks() exception > -- > > Key: HADOOP-12260 > URL: https://issues.apache.org/jira/browse/HADOOP-12260 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.0, 2.7.1 > Environment: OS: CentOS Linux release 7.1.1503 (Core) > Kernel: 3.10.0-229.1.2.el7.x86_64 >Reporter: Marius > > Hi > I was running some streaming jobs with avro files from my hadoop cluster. > They performed poorly so i checked the logs of my datanodes and found this: > http://pastebin.com/DXKJJ55z > The cluster is running on CentOS machines: > CentOS Linux release 7.1.1503 (Core) > This is the Kernel: > 3.10.0-229.1.2.el7.x86_64 > No one on the userlist replied and i could not find anything helpful on the > internet despite disk failure which is unlikely to cause this because here > are several machines and its not very likely that all of their disks fail at > the same time. > This error is not reported on the console when running a job and the error > occurs from time to time and then dissapears and comes back again. > The block size of the cluster is the default value. > This is my command: > hadoop jar hadoop-streaming-2.7.1.jar -files mapper.py,reducer.py,avro-1. > 7.7.jar,avro-mapred-1.7.7-hadoop2.jar -D mapreduce.job.reduces=15 -libjars > avro-1.7.7.jar,avro-mapred-1.7.7-hadoop2.jar -input /Y/Y1.avro -output > /htest/output -mapper mapper.py -reducer reducer.py -inputformat > org.apache.avro.mapred.AvroAsTextInputFormat > Marius -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11505) hadoop-mapreduce-client-nativetask uses bswap where be32toh is needed, doesn't work on non-x86
[ https://issues.apache.org/jira/browse/HADOOP-11505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Burlison updated HADOOP-11505: --- Attachment: HADOOP-11505.004.patch Removed more little-endian code > hadoop-mapreduce-client-nativetask uses bswap where be32toh is needed, > doesn't work on non-x86 > -- > > Key: HADOOP-11505 > URL: https://issues.apache.org/jira/browse/HADOOP-11505 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Labels: BB2015-05-TBR > Attachments: HADOOP-11505.001.patch, HADOOP-11505.003.patch, > HADOOP-11505.004.patch > > > hadoop-mapreduce-client-nativetask fails to use x86 optimizations in some > cases. Also, on some alternate, non-x86, non-ARM architectures the generated > code is incorrect. Thanks to Steve Loughran and Edward Nevill for finding > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up are 64-bit
Alan Burlison created HADOOP-12261: -- Summary: Surefire needs to make sure the JVMs it fires up are 64-bit Key: HADOOP-12261 URL: https://issues.apache.org/jira/browse/HADOOP-12261 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 2.7.1 Reporter: Alan Burlison hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by default and tests fail to start as a result. "-d64" should be added to the command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up are 64-bit
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Burlison reassigned HADOOP-12261: -- Assignee: Alan Burlison > Surefire needs to make sure the JVMs it fires up are 64-bit > --- > > Key: HADOOP-12261 > URL: https://issues.apache.org/jira/browse/HADOOP-12261 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Affects Versions: 2.7.1 >Reporter: Alan Burlison >Assignee: Alan Burlison > > hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include > -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on > platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by > default and tests fail to start as a result. "-d64" should be added to the > command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up are 64-bit
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Burlison updated HADOOP-12261: --- Issue Type: Sub-task (was: Bug) Parent: HADOOP-11985 > Surefire needs to make sure the JVMs it fires up are 64-bit > --- > > Key: HADOOP-12261 > URL: https://issues.apache.org/jira/browse/HADOOP-12261 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Affects Versions: 2.7.1 >Reporter: Alan Burlison > > hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include > -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on > platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by > default and tests fail to start as a result. "-d64" should be added to the > command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12239) StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB
[
https://issues.apache.org/jira/browse/HADOOP-12239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638920#comment-14638920
]
Hudson commented on HADOOP-12239:
-
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #254 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/254/])
HADOOP-12239. StorageException complaining " no lease ID" when updating
FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev
efa97243ecb84b3b468e732897cd685e3869f480)
*
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
* hadoop-common-project/hadoop-common/CHANGES.txt
> StorageException complaining " no lease ID" when updating
> FolderLastModifiedTime in WASB
>
>
> Key: HADOOP-12239
> URL: https://issues.apache.org/jira/browse/HADOOP-12239
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, tools
>Affects Versions: 2.7.0
> Environment: Windows Azure
>Reporter: Duo Xu
>Assignee: Duo Xu
> Fix For: 2.8.0
>
> Attachments: HADOOP-12239.01.patch
>
>
> This is a similar issue as HADOOP-11523 and HADOOP-12089, which I found in a
> customer's HBase cluster logs, but the piece of code is in a different place.
> {code}
> 2015-07-09 13:38:57,388 INFO org.apache.hadoop.hbase.master.SplitLogManager:
> dead splitlog workers
> [workernode3.xxx.b6.internal.cloudapp.net,60020,1436448555180]
> 2015-07-09 13:38:57,466 ERROR org.apache.hadoop.hbase.executor.EventHandler:
> Caught throwable while processing event M_SERVER_SHUTDOWN
> java.io.IOException: failed log splitting for
> workernode12.xxx.b6.internal.cloudapp.net,60020,1436 448566374, will retry
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:343)
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:211)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unable to write RenamePending file for folder
> rename from
> hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374 to
> hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374-splitting
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:258)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.prepareAtomicFolderRename(NativeAzureFileSystem.java:2110)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1998)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.getLogDirs(MasterFileSystem.java:325)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:412)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:390)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:288)
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:204)
> ... 4 more
> Caused by: org.apache.hadoop.fs.azure.AzureException:
> com.microsoft.azure.storage.StorageException: There is currently a lease on
> the blob and no lease ID was specified in the request.
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2598)
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2609)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1366)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1195)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:775)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:255)
> ... 11 more
> Caused by: com.microsoft.azure.storage.StorageException: There is currently a
> lease on the blob and no lease ID was specified in the request.
> at
> com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
> at
> com.microsoft.azure.storage.core.
[jira] [Commented] (HADOOP-12184) Remove unused Linux-specific constants in NativeIO
[ https://issues.apache.org/jira/browse/HADOOP-12184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638919#comment-14638919 ] Hudson commented on HADOOP-12184: - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #254 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/254/]) HADOOP-12184. Remove unused Linux-specific constants in NativeIO (Martin Walsh via Colin P. McCabe) (cmccabe: rev 1b3bceb58c8e536a75fa3f99cc3ceeaba91a07de) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java * hadoop-common-project/hadoop-common/CHANGES.txt > Remove unused Linux-specific constants in NativeIO > -- > > Key: HADOOP-12184 > URL: https://issues.apache.org/jira/browse/HADOOP-12184 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: 2.7.0 > Environment: Solaris >Reporter: Martin Walsh >Assignee: Martin Walsh > Fix For: 2.8.0 > > Attachments: HADOOP-12184.001.patch > > > The POSIX subclass in NativeIO contains the following constants that are not > defined in POSIX: > O_ASYNC, O_FSYNC, O_NDELAY > Stat.S_IFWHT > These constants are not used within Hadoop, as far as I could tell. As they > are not part of the POSIX specification, they should be removed as part of an > effort to create portable code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up are 64-bit
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638936#comment-14638936 ] Allen Wittenauer commented on HADOOP-12261: --- There's actually some speculation that 4gb is too much heap for the surefire plugins. > Surefire needs to make sure the JVMs it fires up are 64-bit > --- > > Key: HADOOP-12261 > URL: https://issues.apache.org/jira/browse/HADOOP-12261 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Affects Versions: 2.7.1 >Reporter: Alan Burlison >Assignee: Alan Burlison > > hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include > -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on > platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by > default and tests fail to start as a result. "-d64" should be added to the > command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12262) deal with "Pending Closed" in releasedocmaker
Allen Wittenauer created HADOOP-12262: - Summary: deal with "Pending Closed" in releasedocmaker Key: HADOOP-12262 URL: https://issues.apache.org/jira/browse/HADOOP-12262 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HADOOP-12111 Reporter: Allen Wittenauer YARN-3641 was set into "Pending Closed" state. 2.7.1 was then marked as Closed, preventing all future updates to that JIRA. There are two things we could do here from a releasedocmaker perspective: a) provide a hook into lint mode that warns about Pending Closed b) provide a flag that treats Pending Closed the same as Fixed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12204) releasedocmaker should pass pylint
[ https://issues.apache.org/jira/browse/HADOOP-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638975#comment-14638975 ] Allen Wittenauer commented on HADOOP-12204: --- Now that we've got pylint support, we should really just fix all of those problems, not just the indentation issue. Updated summary. > releasedocmaker should pass pylint > -- > > Key: HADOOP-12204 > URL: https://issues.apache.org/jira/browse/HADOOP-12204 > Project: Hadoop Common > Issue Type: Sub-task > Components: yetus >Reporter: Chris Nauroth >Priority: Minor > > releasedocmaker.py currently uses a mix of 2-space and 4-space indentation. > It would be good to make this consistent for readability. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12204) releasedocmaker should pass pylint
[ https://issues.apache.org/jira/browse/HADOOP-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-12204: -- Summary: releasedocmaker should pass pylint (was: Use consistent indentation in releasedocmaker.py.) > releasedocmaker should pass pylint > -- > > Key: HADOOP-12204 > URL: https://issues.apache.org/jira/browse/HADOOP-12204 > Project: Hadoop Common > Issue Type: Sub-task > Components: yetus >Reporter: Chris Nauroth >Priority: Minor > > releasedocmaker.py currently uses a mix of 2-space and 4-space indentation. > It would be good to make this consistent for readability. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12256) add support for ruby-lint
[ https://issues.apache.org/jira/browse/HADOOP-12256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-12256: -- Summary: add support for ruby-lint (was: we should add support for ruby-lint) > add support for ruby-lint > - > > Key: HADOOP-12256 > URL: https://issues.apache.org/jira/browse/HADOOP-12256 > Project: Hadoop Common > Issue Type: Sub-task > Components: yetus >Affects Versions: HADOOP-12111 >Reporter: Allen Wittenauer > > We should add support for ruby-lint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12239) StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB
[
https://issues.apache.org/jira/browse/HADOOP-12239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639002#comment-14639002
]
Hudson commented on HADOOP-12239:
-
SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #262 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/262/])
HADOOP-12239. StorageException complaining " no lease ID" when updating
FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev
efa97243ecb84b3b468e732897cd685e3869f480)
* hadoop-common-project/hadoop-common/CHANGES.txt
*
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
> StorageException complaining " no lease ID" when updating
> FolderLastModifiedTime in WASB
>
>
> Key: HADOOP-12239
> URL: https://issues.apache.org/jira/browse/HADOOP-12239
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, tools
>Affects Versions: 2.7.0
> Environment: Windows Azure
>Reporter: Duo Xu
>Assignee: Duo Xu
> Fix For: 2.8.0
>
> Attachments: HADOOP-12239.01.patch
>
>
> This is a similar issue as HADOOP-11523 and HADOOP-12089, which I found in a
> customer's HBase cluster logs, but the piece of code is in a different place.
> {code}
> 2015-07-09 13:38:57,388 INFO org.apache.hadoop.hbase.master.SplitLogManager:
> dead splitlog workers
> [workernode3.xxx.b6.internal.cloudapp.net,60020,1436448555180]
> 2015-07-09 13:38:57,466 ERROR org.apache.hadoop.hbase.executor.EventHandler:
> Caught throwable while processing event M_SERVER_SHUTDOWN
> java.io.IOException: failed log splitting for
> workernode12.xxx.b6.internal.cloudapp.net,60020,1436 448566374, will retry
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:343)
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:211)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unable to write RenamePending file for folder
> rename from
> hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374 to
> hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374-splitting
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:258)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.prepareAtomicFolderRename(NativeAzureFileSystem.java:2110)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1998)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.getLogDirs(MasterFileSystem.java:325)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:412)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:390)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:288)
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:204)
> ... 4 more
> Caused by: org.apache.hadoop.fs.azure.AzureException:
> com.microsoft.azure.storage.StorageException: There is currently a lease on
> the blob and no lease ID was specified in the request.
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2598)
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2609)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1366)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1195)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:775)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:255)
> ... 11 more
> Caused by: com.microsoft.azure.storage.StorageException: There is currently a
> lease on the blob and no lease ID was specified in the request.
> at
> com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
> at
> com.microsoft.azure.sto
[jira] [Created] (HADOOP-12263) TestAuth KerberosAuthenticator test consistently fails with IBM JVM on some platforms
Tony Reix created HADOOP-12263: -- Summary: TestAuth KerberosAuthenticator test consistently fails with IBM JVM on some platforms Key: HADOOP-12263 URL: https://issues.apache.org/jira/browse/HADOOP-12263 Project: Hadoop Common Issue Type: Bug Components: security, test Affects Versions: 2.7.1, 2.6.0 Environment: IBM JVM on AIX and RHEL7/PPC64LE. Not issue with IBM JVM on Ubuntu/x86_64 . Reporter: Tony Reix Test TestKerberosAuthenticator consistently fails on platforms AIX and RHEL7/PPC64LE with IBM JVM though it works fine on Ubuntu/x86_64/IBMJVM and on RHEL7/PPC64LE with OpenJDK. Test is: mvn test -Dtest=org.apache.hadoop.security.authentication.client.TestKerberosAuthenticator Error is: 2015-07-23 09:23:34,226 ERROR KerberosProtocolHandler - /127.0.0.1:44353 EXCEPTION org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 00 8B) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627) at java.lang.Thread.run(Thread.java:809) Caused by: java.lang.NullPointerException: message at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) ... 15 more This looks close to JIRAs: https://issues.apache.org/jira/browse/DIRSERVER-2071 https://issues.apache.org/jira/browse/HADOOP-12090 http://qnalist.com/questions/6070807/a-slew-of-mini-kdc-related-unit-test-failures -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12184) Remove unused Linux-specific constants in NativeIO
[ https://issues.apache.org/jira/browse/HADOOP-12184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639001#comment-14639001 ] Hudson commented on HADOOP-12184: - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #262 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/262/]) HADOOP-12184. Remove unused Linux-specific constants in NativeIO (Martin Walsh via Colin P. McCabe) (cmccabe: rev 1b3bceb58c8e536a75fa3f99cc3ceeaba91a07de) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java * hadoop-common-project/hadoop-common/CHANGES.txt > Remove unused Linux-specific constants in NativeIO > -- > > Key: HADOOP-12184 > URL: https://issues.apache.org/jira/browse/HADOOP-12184 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: 2.7.0 > Environment: Solaris >Reporter: Martin Walsh >Assignee: Martin Walsh > Fix For: 2.8.0 > > Attachments: HADOOP-12184.001.patch > > > The POSIX subclass in NativeIO contains the following constants that are not > defined in POSIX: > O_ASYNC, O_FSYNC, O_NDELAY > Stat.S_IFWHT > These constants are not used within Hadoop, as far as I could tell. As they > are not part of the POSIX specification, they should be removed as part of an > effort to create portable code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12184) Remove unused Linux-specific constants in NativeIO
[ https://issues.apache.org/jira/browse/HADOOP-12184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639025#comment-14639025 ] Hudson commented on HADOOP-12184: - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2211 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2211/]) HADOOP-12184. Remove unused Linux-specific constants in NativeIO (Martin Walsh via Colin P. McCabe) (cmccabe: rev 1b3bceb58c8e536a75fa3f99cc3ceeaba91a07de) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java * hadoop-common-project/hadoop-common/CHANGES.txt > Remove unused Linux-specific constants in NativeIO > -- > > Key: HADOOP-12184 > URL: https://issues.apache.org/jira/browse/HADOOP-12184 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: 2.7.0 > Environment: Solaris >Reporter: Martin Walsh >Assignee: Martin Walsh > Fix For: 2.8.0 > > Attachments: HADOOP-12184.001.patch > > > The POSIX subclass in NativeIO contains the following constants that are not > defined in POSIX: > O_ASYNC, O_FSYNC, O_NDELAY > Stat.S_IFWHT > These constants are not used within Hadoop, as far as I could tell. As they > are not part of the POSIX specification, they should be removed as part of an > effort to create portable code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12239) StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB
[
https://issues.apache.org/jira/browse/HADOOP-12239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639026#comment-14639026
]
Hudson commented on HADOOP-12239:
-
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2211 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2211/])
HADOOP-12239. StorageException complaining " no lease ID" when updating
FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev
efa97243ecb84b3b468e732897cd685e3869f480)
* hadoop-common-project/hadoop-common/CHANGES.txt
*
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
> StorageException complaining " no lease ID" when updating
> FolderLastModifiedTime in WASB
>
>
> Key: HADOOP-12239
> URL: https://issues.apache.org/jira/browse/HADOOP-12239
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, tools
>Affects Versions: 2.7.0
> Environment: Windows Azure
>Reporter: Duo Xu
>Assignee: Duo Xu
> Fix For: 2.8.0
>
> Attachments: HADOOP-12239.01.patch
>
>
> This is a similar issue as HADOOP-11523 and HADOOP-12089, which I found in a
> customer's HBase cluster logs, but the piece of code is in a different place.
> {code}
> 2015-07-09 13:38:57,388 INFO org.apache.hadoop.hbase.master.SplitLogManager:
> dead splitlog workers
> [workernode3.xxx.b6.internal.cloudapp.net,60020,1436448555180]
> 2015-07-09 13:38:57,466 ERROR org.apache.hadoop.hbase.executor.EventHandler:
> Caught throwable while processing event M_SERVER_SHUTDOWN
> java.io.IOException: failed log splitting for
> workernode12.xxx.b6.internal.cloudapp.net,60020,1436 448566374, will retry
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:343)
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:211)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unable to write RenamePending file for folder
> rename from
> hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374 to
> hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374-splitting
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:258)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.prepareAtomicFolderRename(NativeAzureFileSystem.java:2110)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1998)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.getLogDirs(MasterFileSystem.java:325)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:412)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:390)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:288)
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:204)
> ... 4 more
> Caused by: org.apache.hadoop.fs.azure.AzureException:
> com.microsoft.azure.storage.StorageException: There is currently a lease on
> the blob and no lease ID was specified in the request.
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2598)
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2609)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1366)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1195)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:775)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:255)
> ... 11 more
> Caused by: com.microsoft.azure.storage.StorageException: There is currently a
> lease on the blob and no lease ID was specified in the request.
> at
> com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
> at
> com.microsoft.azure.storage.core.
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up are 64-bit
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639047#comment-14639047 ] Alan Burlison commented on HADOOP-12261: Well, there are two possibilities, either decrease the mem allocation or add -d64, either are trivial to do. > Surefire needs to make sure the JVMs it fires up are 64-bit > --- > > Key: HADOOP-12261 > URL: https://issues.apache.org/jira/browse/HADOOP-12261 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Affects Versions: 2.7.1 >Reporter: Alan Burlison >Assignee: Alan Burlison > > hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include > -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on > platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by > default and tests fail to start as a result. "-d64" should be added to the > command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12263) TestAuth KerberosAuthenticator test consistently fails with IBM JVM on some platforms
[ https://issues.apache.org/jira/browse/HADOOP-12263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tony Reix updated HADOOP-12263: --- Target Version/s: 2.8.0 > TestAuth KerberosAuthenticator test consistently fails with IBM JVM on some > platforms > - > > Key: HADOOP-12263 > URL: https://issues.apache.org/jira/browse/HADOOP-12263 > Project: Hadoop Common > Issue Type: Bug > Components: security, test >Affects Versions: 2.6.0, 2.7.1 > Environment: IBM JVM on AIX and RHEL7/PPC64LE. > Not issue with IBM JVM on Ubuntu/x86_64 . >Reporter: Tony Reix > > Test TestKerberosAuthenticator consistently fails on platforms AIX and > RHEL7/PPC64LE with IBM JVM though it works fine on Ubuntu/x86_64/IBMJVM and > on RHEL7/PPC64LE with OpenJDK. > Test is: > mvn test > -Dtest=org.apache.hadoop.security.authentication.client.TestKerberosAuthenticator > Error is: > 2015-07-23 09:23:34,226 ERROR KerberosProtocolHandler - /127.0.0.1:44353 > EXCEPTION > org.apache.mina.filter.codec.ProtocolDecoderException: > java.lang.NullPointerException: message (Hexdump: 00 00 00 8B) > at > org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234) > at > org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) > at > org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48) > at > org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802) > at > org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120) > at > org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) > at > org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) > at > org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604) > at > org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564) > at > org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553) > at > org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57) > at > org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892) > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627) > at java.lang.Thread.run(Thread.java:809) > Caused by: java.lang.NullPointerException: message > at > org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44) > at > org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65) > at > org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224) > ... 15 more > This looks close to JIRAs: > https://issues.apache.org/jira/browse/DIRSERVER-2071 > https://issues.apache.org/jira/browse/HADOOP-12090 > http://qnalist.com/questions/6070807/a-slew-of-mini-kdc-related-unit-test-failures -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11505) hadoop-mapreduce-client-nativetask uses bswap where be32toh is needed, doesn't work on non-x86
[
https://issues.apache.org/jira/browse/HADOOP-11505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639061#comment-14639061
]
Hadoop QA commented on HADOOP-11505:
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 6s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 6 new or modified test files. |
| {color:green}+1{color} | javac | 7m 40s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 40s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 26s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 20s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 36s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 22m 24s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | mapreduce tests | 10m 35s | Tests passed in
hadoop-mapreduce-client-nativetask. |
| | | 73m 51s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12746805/HADOOP-11505.004.patch
|
| Optional Tests | javac unit javadoc findbugs checkstyle |
| git revision | trunk / ee98d63 |
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7331/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-mapreduce-client-nativetask test log |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7331/artifact/patchprocess/testrun_hadoop-mapreduce-client-nativetask.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7331/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7331/console |
This message was automatically generated.
> hadoop-mapreduce-client-nativetask uses bswap where be32toh is needed,
> doesn't work on non-x86
> --
>
> Key: HADOOP-11505
> URL: https://issues.apache.org/jira/browse/HADOOP-11505
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Labels: BB2015-05-TBR
> Attachments: HADOOP-11505.001.patch, HADOOP-11505.003.patch,
> HADOOP-11505.004.patch
>
>
> hadoop-mapreduce-client-nativetask fails to use x86 optimizations in some
> cases. Also, on some alternate, non-x86, non-ARM architectures the generated
> code is incorrect. Thanks to Steve Loughran and Edward Nevill for finding
> this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12258) Need translate java.nio.file.NoSuchFileException to FileNotFoundException in DeprecatedRawLocalFileStatus constructor
[
https://issues.apache.org/jira/browse/HADOOP-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639082#comment-14639082
]
Steve Loughran commented on HADOOP-12258:
-
This is a major regression —it's good you caught it.
The requirement that getStatus on a nonexistent path throws an FNFE should be
part of the fs contract tests -it should have been caught there.
Similarly, we need a test which checks the requirement path.setTimes(time)
throws an FNFE if the path doesn't exist.
Do you fancy adding a new Abstract FS Contract test for set times, so that we
can do it consistently across a filesystems? The same tests could be used (we
could add an FS granularity option too to the test contract, so that
filesystems could declare tighter vs looser times (I think FAT32 is 2s, for
example)
> Need translate java.nio.file.NoSuchFileException to FileNotFoundException in
> DeprecatedRawLocalFileStatus constructor
> -
>
> Key: HADOOP-12258
> URL: https://issues.apache.org/jira/browse/HADOOP-12258
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Critical
> Attachments: HADOOP-12258.000.patch
>
>
> need translate java.nio.file.NoSuchFileException to FileNotFoundException in
> DeprecatedRawLocalFileStatus constructor.
> HADOOP-12045 adds nio to support access time, but nio will create
> java.nio.file.NoSuchFileException instead of FileNotFoundException.
> many hadoop codes depend on FileNotFoundException to decide whether a file
> exists. for example {{FileContext.util().exists()}}.
> {code}
> public boolean exists(final Path f) throws AccessControlException,
> UnsupportedFileSystemException, IOException {
> try {
> FileStatus fs = FileContext.this.getFileStatus(f);
> assert fs != null;
> return true;
> } catch (FileNotFoundException e) {
> return false;
> }
> }
> {code}
> same for {{FileSystem#exists}}
> {code}
> public boolean exists(Path f) throws IOException {
> try {
> return getFileStatus(f) != null;
> } catch (FileNotFoundException e) {
> return false;
> }
> }
> {code}
> NoSuchFileException will break these functions.
> Since {{exists}} is one of the most used API in FileSystem, this issue is
> very critical.
> Several test failures for TestDeletionService are caused by this issue:
> https://builds.apache.org/job/PreCommit-YARN-Build/8630/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testRelativeDelete/
> https://builds.apache.org/job/PreCommit-YARN-Build/8632/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testAbsDelete/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12260) BlockSender.sendChunks() exception
[
https://issues.apache.org/jira/browse/HADOOP-12260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639146#comment-14639146
]
zhihai xu commented on HADOOP-12260:
Hi [~butzy92], thanks for reporting this issue.
{code}
2015-07-17 16:33:45,671 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
BlockSender.sendChunks() exception:
java.io.IOException: Die Verbindung wurde vom Kommunikationspartner
zurückgesetzt
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at
sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:443)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:575)
at
org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:559)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:728)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:496)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:116)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
at java.lang.Thread.run(Thread.java:745)
{code}
Base on above stack trace, This looks like a HDFS issue, updated the component
to HDFS.
> BlockSender.sendChunks() exception
> --
>
> Key: HADOOP-12260
> URL: https://issues.apache.org/jira/browse/HADOOP-12260
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.1
> Environment: OS: CentOS Linux release 7.1.1503 (Core)
> Kernel: 3.10.0-229.1.2.el7.x86_64
>Reporter: Marius
>
> Hi
> I was running some streaming jobs with avro files from my hadoop cluster.
> They performed poorly so i checked the logs of my datanodes and found this:
> http://pastebin.com/DXKJJ55z
> The cluster is running on CentOS machines:
> CentOS Linux release 7.1.1503 (Core)
> This is the Kernel:
> 3.10.0-229.1.2.el7.x86_64
> No one on the userlist replied and i could not find anything helpful on the
> internet despite disk failure which is unlikely to cause this because here
> are several machines and its not very likely that all of their disks fail at
> the same time.
> This error is not reported on the console when running a job and the error
> occurs from time to time and then dissapears and comes back again.
> The block size of the cluster is the default value.
> This is my command:
> hadoop jar hadoop-streaming-2.7.1.jar -files mapper.py,reducer.py,avro-1.
> 7.7.jar,avro-mapred-1.7.7-hadoop2.jar -D mapreduce.job.reduces=15 -libjars
> avro-1.7.7.jar,avro-mapred-1.7.7-hadoop2.jar -input /Y/Y1.avro -output
> /htest/output -mapper mapper.py -reducer reducer.py -inputformat
> org.apache.avro.mapred.AvroAsTextInputFormat
> Marius
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12161) Add getStoragePolicy API to the FileSystem interface
[
https://issues.apache.org/jira/browse/HADOOP-12161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-12161:
---
Issue Type: Improvement (was: Bug)
> Add getStoragePolicy API to the FileSystem interface
>
>
> Key: HADOOP-12161
> URL: https://issues.apache.org/jira/browse/HADOOP-12161
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HADOOP-12161-001.patch, HADOOP-12161-002.patch,
> HADOOP-12161-003.patch, HADOOP-12161-004.patch
>
>
> HDFS-8345 added {{FileSystem#getAllStoragePolicies}} and
> {{FileSystem#setStoragePolicy}}. Jira to
> # Add a corresponding {{FileSystem#getStoragePolicy}} to query the storage
> policy for a given file/directory.
> # Add corresponding implementation for HDFS i.e.
> {{DistributedFileSystem#getStoragePolicy}}.
> # Update the [FileSystem
> specification|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/filesystem.html].
> This will require editing
> _hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md_.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12161) Add getStoragePolicy API to the FileSystem interface
[
https://issues.apache.org/jira/browse/HADOOP-12161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-12161:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Target Version/s: (was: 2.8.0)
Status: Resolved (was: Patch Available)
I committed it for 2.8.0.
Thanks a lot for the reviews and feedback Jing and Vinayakumar. We can either
cache policies on the client or add a new getStoragePolicy RPC to the NN as
Vinayakumar suggested in a followup task.
> Add getStoragePolicy API to the FileSystem interface
>
>
> Key: HADOOP-12161
> URL: https://issues.apache.org/jira/browse/HADOOP-12161
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Arpit Agarwal
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HADOOP-12161-001.patch, HADOOP-12161-002.patch,
> HADOOP-12161-003.patch, HADOOP-12161-004.patch
>
>
> HDFS-8345 added {{FileSystem#getAllStoragePolicies}} and
> {{FileSystem#setStoragePolicy}}. Jira to
> # Add a corresponding {{FileSystem#getStoragePolicy}} to query the storage
> policy for a given file/directory.
> # Add corresponding implementation for HDFS i.e.
> {{DistributedFileSystem#getStoragePolicy}}.
> # Update the [FileSystem
> specification|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/filesystem.html].
> This will require editing
> _hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md_.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12045) Enable LocalFileSystem#setTimes to change atime
[ https://issues.apache.org/jira/browse/HADOOP-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639165#comment-14639165 ] Chris Nauroth commented on HADOOP-12045: Hi [~fjk]. FYI, there is a follow-up bug fix to this being discussed on HADOOP-12258. > Enable LocalFileSystem#setTimes to change atime > --- > > Key: HADOOP-12045 > URL: https://issues.apache.org/jira/browse/HADOOP-12045 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Kazuho Fujii >Assignee: Kazuho Fujii >Priority: Minor > Fix For: 2.8.0 > > Attachments: HADOOP-12045.001.patch, HADOOP-12045.002.patch, > HADOOP-12045.003.patch, HADOOP-12045.004-1.patch, HADOOP-12045.004-2.patch, > HADOOP-12045.005-1.patch, HADOOP-12045.005-2.patch, HADOOP-12045.006.patch > > > LocalFileSystem#setTimes method can not change the last access time currently. > With java.nio.file package in Java 7, we can implement the function easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12161) Add getStoragePolicy API to the FileSystem interface
[
https://issues.apache.org/jira/browse/HADOOP-12161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639167#comment-14639167
]
Arpit Agarwal commented on HADOOP-12161:
Filed HDFS-8815 for followup work. Documentation can be fixed as part of
HADOOP-11981.
> Add getStoragePolicy API to the FileSystem interface
>
>
> Key: HADOOP-12161
> URL: https://issues.apache.org/jira/browse/HADOOP-12161
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HADOOP-12161-001.patch, HADOOP-12161-002.patch,
> HADOOP-12161-003.patch, HADOOP-12161-004.patch
>
>
> HDFS-8345 added {{FileSystem#getAllStoragePolicies}} and
> {{FileSystem#setStoragePolicy}}. Jira to
> # Add a corresponding {{FileSystem#getStoragePolicy}} to query the storage
> policy for a given file/directory.
> # Add corresponding implementation for HDFS i.e.
> {{DistributedFileSystem#getStoragePolicy}}.
> # Update the [FileSystem
> specification|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/filesystem.html].
> This will require editing
> _hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md_.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12258) Need translate java.nio.file.NoSuchFileException to FileNotFoundException in DeprecatedRawLocalFileStatus constructor
[
https://issues.apache.org/jira/browse/HADOOP-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639170#comment-14639170
]
Chris Nauroth commented on HADOOP-12258:
[~zxu], thank you for spotting this and filing the bug report. I agree with
Steve that a contract test would be great for this.
> Need translate java.nio.file.NoSuchFileException to FileNotFoundException in
> DeprecatedRawLocalFileStatus constructor
> -
>
> Key: HADOOP-12258
> URL: https://issues.apache.org/jira/browse/HADOOP-12258
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Critical
> Attachments: HADOOP-12258.000.patch
>
>
> need translate java.nio.file.NoSuchFileException to FileNotFoundException in
> DeprecatedRawLocalFileStatus constructor.
> HADOOP-12045 adds nio to support access time, but nio will create
> java.nio.file.NoSuchFileException instead of FileNotFoundException.
> many hadoop codes depend on FileNotFoundException to decide whether a file
> exists. for example {{FileContext.util().exists()}}.
> {code}
> public boolean exists(final Path f) throws AccessControlException,
> UnsupportedFileSystemException, IOException {
> try {
> FileStatus fs = FileContext.this.getFileStatus(f);
> assert fs != null;
> return true;
> } catch (FileNotFoundException e) {
> return false;
> }
> }
> {code}
> same for {{FileSystem#exists}}
> {code}
> public boolean exists(Path f) throws IOException {
> try {
> return getFileStatus(f) != null;
> } catch (FileNotFoundException e) {
> return false;
> }
> }
> {code}
> NoSuchFileException will break these functions.
> Since {{exists}} is one of the most used API in FileSystem, this issue is
> very critical.
> Several test failures for TestDeletionService are caused by this issue:
> https://builds.apache.org/job/PreCommit-YARN-Build/8630/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testRelativeDelete/
> https://builds.apache.org/job/PreCommit-YARN-Build/8632/testReport/org.apache.hadoop.yarn.server.nodemanager/TestDeletionService/testAbsDelete/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12161) Add getStoragePolicy API to the FileSystem interface
[
https://issues.apache.org/jira/browse/HADOOP-12161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639175#comment-14639175
]
Hudson commented on HADOOP-12161:
-
FAILURE: Integrated in Hadoop-trunk-Commit #8206 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8206/])
HADOOP-12161. Add getStoragePolicy API to the FileSystem interface.
(Contributed by Brahma Reddy Battula) (arp: rev
adfa34ff9992295a6d2496b259d8c483ed90b566)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestHarFileSystem.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFs.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/Hdfs.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java
* hadoop-common-project/hadoop-common/CHANGES.txt
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFs.java
> Add getStoragePolicy API to the FileSystem interface
>
>
> Key: HADOOP-12161
> URL: https://issues.apache.org/jira/browse/HADOOP-12161
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HADOOP-12161-001.patch, HADOOP-12161-002.patch,
> HADOOP-12161-003.patch, HADOOP-12161-004.patch
>
>
> HDFS-8345 added {{FileSystem#getAllStoragePolicies}} and
> {{FileSystem#setStoragePolicy}}. Jira to
> # Add a corresponding {{FileSystem#getStoragePolicy}} to query the storage
> policy for a given file/directory.
> # Add corresponding implementation for HDFS i.e.
> {{DistributedFileSystem#getStoragePolicy}}.
> # Update the [FileSystem
> specification|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/filesystem.html].
> This will require editing
> _hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md_.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12161) Add getStoragePolicy API to the FileSystem interface
[
https://issues.apache.org/jira/browse/HADOOP-12161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639228#comment-14639228
]
Brahma Reddy Battula commented on HADOOP-12161:
---
[~arpitagarwal] thanks a lot for review and committing and thanks to
[~jingzhao] and [~vinayrpet] for additional review.
Agree to handle to follow up Jira . Will raise for same.
> Add getStoragePolicy API to the FileSystem interface
>
>
> Key: HADOOP-12161
> URL: https://issues.apache.org/jira/browse/HADOOP-12161
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HADOOP-12161-001.patch, HADOOP-12161-002.patch,
> HADOOP-12161-003.patch, HADOOP-12161-004.patch
>
>
> HDFS-8345 added {{FileSystem#getAllStoragePolicies}} and
> {{FileSystem#setStoragePolicy}}. Jira to
> # Add a corresponding {{FileSystem#getStoragePolicy}} to query the storage
> policy for a given file/directory.
> # Add corresponding implementation for HDFS i.e.
> {{DistributedFileSystem#getStoragePolicy}}.
> # Update the [FileSystem
> specification|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/filesystem.html].
> This will require editing
> _hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md_.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12247) Convert 'unit' to 'junit'
[ https://issues.apache.org/jira/browse/HADOOP-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-12247: -- Labels: (was: newbie) > Convert 'unit' to 'junit' > - > > Key: HADOOP-12247 > URL: https://issues.apache.org/jira/browse/HADOOP-12247 > Project: Hadoop Common > Issue Type: Sub-task > Components: yetus >Affects Versions: HADOOP-12111 >Reporter: Allen Wittenauer > > In order to support other unit test systems, we should convert 'unit' to be > specifically 'junit' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12247) Convert 'unit' to 'junit'
[ https://issues.apache.org/jira/browse/HADOOP-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-12247: -- Status: Patch Available (was: Open) > Convert 'unit' to 'junit' > - > > Key: HADOOP-12247 > URL: https://issues.apache.org/jira/browse/HADOOP-12247 > Project: Hadoop Common > Issue Type: Sub-task > Components: yetus >Affects Versions: HADOOP-12111 >Reporter: Allen Wittenauer > Attachments: HADOOP-12247.HADOOP-12111.00.patch > > > In order to support other unit test systems, we should convert 'unit' to be > specifically 'junit' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12247) Convert 'unit' to 'junit'
[ https://issues.apache.org/jira/browse/HADOOP-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-12247: -- Attachment: HADOOP-12247.HADOOP-12111.00.patch -00: * make unit test file formats pluggable * move junit to it a plugin > Convert 'unit' to 'junit' > - > > Key: HADOOP-12247 > URL: https://issues.apache.org/jira/browse/HADOOP-12247 > Project: Hadoop Common > Issue Type: Sub-task > Components: yetus >Affects Versions: HADOOP-12111 >Reporter: Allen Wittenauer > Attachments: HADOOP-12247.HADOOP-12111.00.patch > > > In order to support other unit test systems, we should convert 'unit' to be > specifically 'junit' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HADOOP-12247) Convert 'unit' to 'junit'
[ https://issues.apache.org/jira/browse/HADOOP-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer reassigned HADOOP-12247: - Assignee: Allen Wittenauer > Convert 'unit' to 'junit' > - > > Key: HADOOP-12247 > URL: https://issues.apache.org/jira/browse/HADOOP-12247 > Project: Hadoop Common > Issue Type: Sub-task > Components: yetus >Affects Versions: HADOOP-12111 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-12247.HADOOP-12111.00.patch > > > In order to support other unit test systems, we should convert 'unit' to be > specifically 'junit' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HADOOP-12247) Convert 'unit' to 'junit'
[ https://issues.apache.org/jira/browse/HADOOP-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639246#comment-14639246 ] Allen Wittenauer edited comment on HADOOP-12247 at 7/23/15 5:44 PM: -00: * make unit test file formats pluggable * move junit to a plugin was (Author: aw): -00: * make unit test file formats pluggable * move junit to it a plugin > Convert 'unit' to 'junit' > - > > Key: HADOOP-12247 > URL: https://issues.apache.org/jira/browse/HADOOP-12247 > Project: Hadoop Common > Issue Type: Sub-task > Components: yetus >Affects Versions: HADOOP-12111 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-12247.HADOOP-12111.00.patch > > > In order to support other unit test systems, we should convert 'unit' to be > specifically 'junit' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12247) Convert 'unit' to 'junit'
[ https://issues.apache.org/jira/browse/HADOOP-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639258#comment-14639258 ] Hadoop QA commented on HADOOP-12247: (!) A patch to the files used for the QA process has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7332/console in case of problems. > Convert 'unit' to 'junit' > - > > Key: HADOOP-12247 > URL: https://issues.apache.org/jira/browse/HADOOP-12247 > Project: Hadoop Common > Issue Type: Sub-task > Components: yetus >Affects Versions: HADOOP-12111 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-12247.HADOOP-12111.00.patch > > > In order to support other unit test systems, we should convert 'unit' to be > specifically 'junit' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12247) Convert 'unit' to 'junit'
[
https://issues.apache.org/jira/browse/HADOOP-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639259#comment-14639259
]
Hadoop QA commented on HADOOP-12247:
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} precommit patch detected. {color} |
| {color:blue}0{color} | {color:blue} @author {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipping @author checks as test-patch.sh has been
patched. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m 10s
{color} | {color:red} The applied patch generated 3 new shellcheck (v0.3.3)
issues (total was 24, now 27). {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 0m 28s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12746829/HADOOP-12247.HADOOP-12111.00.patch
|
| git revision | HADOOP-12111 / 1e4f361 |
| Optional Tests | asflicense shellcheck |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/dev-support-test/personality/hadoop.sh
|
| Default Java | 1.7.0_55 |
| Multi-JDK versions | /home/jenkins/tools/java/jdk1.8.0:1.8.0
/home/jenkins/tools/java/jdk1.7.0_55:1.7.0_55 |
| shellcheck | v0.3.3 (This is an old version that has serious bugs. Consider
upgrading.) |
| shellcheck |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7332/artifact/patchprocess/diffpatchshellcheck.txt
|
| Max memory used | 48MB |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7332/console |
This message was automatically generated.
> Convert 'unit' to 'junit'
> -
>
> Key: HADOOP-12247
> URL: https://issues.apache.org/jira/browse/HADOOP-12247
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: yetus
>Affects Versions: HADOOP-12111
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
> Attachments: HADOOP-12247.HADOOP-12111.00.patch
>
>
> In order to support other unit test systems, we should convert 'unit' to be
> specifically 'junit'
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up are 64-bit
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639263#comment-14639263 ] Colin Patrick McCabe commented on HADOOP-12261: --- On a 32-bit machine with a traditional 50/50 kernel/usermode split, 2 GB is the most any process can have. Are we comfortable cutting the surefire plugin memory reservation to that level? If not, I think it might be time to wave goodbye to 32-bit... I don't think anyone is really running Hadoop on 32-bit. Even ARM has gone 64-bit. > Surefire needs to make sure the JVMs it fires up are 64-bit > --- > > Key: HADOOP-12261 > URL: https://issues.apache.org/jira/browse/HADOOP-12261 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Affects Versions: 2.7.1 >Reporter: Alan Burlison >Assignee: Alan Burlison > > hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include > -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on > platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by > default and tests fail to start as a result. "-d64" should be added to the > command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12261) Surefire needs to make sure the JVMs it fires up are 64-bit
[ https://issues.apache.org/jira/browse/HADOOP-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639268#comment-14639268 ] Allen Wittenauer commented on HADOOP-12261: --- bq. I think it might be time to wave goodbye to 32-bit That'd be an incompatible change... > Surefire needs to make sure the JVMs it fires up are 64-bit > --- > > Key: HADOOP-12261 > URL: https://issues.apache.org/jira/browse/HADOOP-12261 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Affects Versions: 2.7.1 >Reporter: Alan Burlison >Assignee: Alan Burlison > > hadoop-project/pom.xml sets maven-surefire-plugin.argLine to include > -Xmx4096m. Allocating that amount of memory requires a 64-bit JVM, but on > platforms with both 32 and 64-bit JVMs surefire runs the 32 bit version by > default and tests fail to start as a result. "-d64" should be added to the > command-line arguments to ensure a 64-bit JVM is always used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12259) Utility to Dynamic port allocation
[ https://issues.apache.org/jira/browse/HADOOP-12259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639355#comment-14639355 ] Robert Kanter commented on HADOOP-12259: LGTM +1 [~varun_saxena], any more comments? > Utility to Dynamic port allocation > -- > > Key: HADOOP-12259 > URL: https://issues.apache.org/jira/browse/HADOOP-12259 > Project: Hadoop Common > Issue Type: Bug > Components: test, util >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HADOOP-12259.patch > > > As per discussion in YARN-3528 and [~rkanter] comment [here | > https://issues.apache.org/jira/browse/YARN-3528?focusedCommentId=14637700&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14637700 > ] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12259) Utility to Dynamic port allocation
[ https://issues.apache.org/jira/browse/HADOOP-12259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639381#comment-14639381 ] Varun Saxena commented on HADOOP-12259: --- +1 LGTM too > Utility to Dynamic port allocation > -- > > Key: HADOOP-12259 > URL: https://issues.apache.org/jira/browse/HADOOP-12259 > Project: Hadoop Common > Issue Type: Bug > Components: test, util >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HADOOP-12259.patch > > > As per discussion in YARN-3528 and [~rkanter] comment [here | > https://issues.apache.org/jira/browse/YARN-3528?focusedCommentId=14637700&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14637700 > ] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10854) unit tests for the shell scripts
[ https://issues.apache.org/jira/browse/HADOOP-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639474#comment-14639474 ] Allen Wittenauer commented on HADOOP-10854: --- So, this *only* executes if -Dskiptests is on the command line. I'm at a loss as to why though. :( > unit tests for the shell scripts > > > Key: HADOOP-10854 > URL: https://issues.apache.org/jira/browse/HADOOP-10854 > Project: Hadoop Common > Issue Type: Test > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Labels: scripts > Attachments: HADOOP-10854.00.patch, HADOOP-10854.01.patch, > HADOOP-10854.02.patch, HADOOP-10854.03.patch > > > With HADOOP-9902 moving a lot of the core functionality to functions, we > should build some unit tests for them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10854) unit tests for the shell scripts
[
https://issues.apache.org/jira/browse/HADOOP-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639497#comment-14639497
]
Sean Busbey commented on HADOOP-10854:
--
just to confirm before I try to reproduce, it only runs when the command line
is like {{mvn -Pshelltest -DskipTests clean test}}?
> unit tests for the shell scripts
>
>
> Key: HADOOP-10854
> URL: https://issues.apache.org/jira/browse/HADOOP-10854
> Project: Hadoop Common
> Issue Type: Test
> Components: scripts
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
> Labels: scripts
> Attachments: HADOOP-10854.00.patch, HADOOP-10854.01.patch,
> HADOOP-10854.02.patch, HADOOP-10854.03.patch
>
>
> With HADOOP-9902 moving a lot of the core functionality to functions, we
> should build some unit tests for them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11731) Rework the changelog and releasenotes
[ https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-11731: - Affects Version/s: (was: 3.0.0) 2.7.0 Target Version/s: (was: 3.0.0) Fix Version/s: 2.8.0 I just backported this to branch-2, just a minor conflict in site.xml I fixed up. Thanks again to Allen for working on this. > Rework the changelog and releasenotes > - > > Key: HADOOP-11731 > URL: https://issues.apache.org/jira/browse/HADOOP-11731 > Project: Hadoop Common > Issue Type: New Feature > Components: documentation >Affects Versions: 2.7.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Fix For: 2.8.0, 3.0.0 > > Attachments: HADOOP-11731-00.patch, HADOOP-11731-01.patch, > HADOOP-11731-03.patch, HADOOP-11731-04.patch, HADOOP-11731-05.patch, > HADOOP-11731-06.patch, HADOOP-11731-07.patch > > > The current way we generate these build artifacts is awful. Plus they are > ugly and, in the case of release notes, very hard to pick out what is > important. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11792) Remove all of the CHANGES.txt files
[
https://issues.apache.org/jira/browse/HADOOP-11792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639508#comment-14639508
]
Andrew Wang commented on HADOOP-11792:
--
Based on the positive mailing list discussion, I'm just going to remove
CHANGES.txt from trunk and branch-2 unless there are other objections. I don't
plan to post a patch, since CHANGES.txt changes have never gone through Jenkins
or required a +1. It'll literally be running {{git rm **/CHANGES.txt}} and
committing it referencing this JIRA.
> Remove all of the CHANGES.txt files
> ---
>
> Key: HADOOP-11792
> URL: https://issues.apache.org/jira/browse/HADOOP-11792
> Project: Hadoop Common
> Issue Type: Task
> Components: build
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
>Assignee: Jayaradha
>
> With the commit of HADOOP-11731, the CHANGES.txt files are now EOLed. We
> should remove them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12264) Update create-release.sh to pass -Preleasedocs
Andrew Wang created HADOOP-12264: Summary: Update create-release.sh to pass -Preleasedocs Key: HADOOP-12264 URL: https://issues.apache.org/jira/browse/HADOOP-12264 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.8.0 Reporter: Andrew Wang Assignee: Andrew Wang We use the create-release.sh script to create the release artifacts. With the way of autogenerating CHANGES.txt we need to update the script to also pass "-Preleasedocs". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12189) improve CallQueueManager#swapQueue to
[
https://issues.apache.org/jira/browse/HADOOP-12189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated HADOOP-12189:
---
Summary: improve CallQueueManager#swapQueue to (was: CallQueueManager may
drop elements from the queue sometimes when calling swapQueue)
> improve CallQueueManager#swapQueue to
> --
>
> Key: HADOOP-12189
> URL: https://issues.apache.org/jira/browse/HADOOP-12189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc, test
>Affects Versions: 2.7.1
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: HADOOP-12189.000.patch, HADOOP-12189.001.patch,
> HADOOP-12189.none_guarantee.000.patch, HADOOP-12189.none_guarantee.001.patch,
> HADOOP-12189.none_guarantee.002.patch
>
>
> CallQueueManager may drop elements from the queue sometimes when calling
> {{swapQueue}}.
> The following test failure from TestCallQueueManager shown some elements in
> the queue are dropped.
> https://builds.apache.org/job/PreCommit-HADOOP-Build/7150/testReport/org.apache.hadoop.ipc/TestCallQueueManager/testSwapUnderContention/
> {code}
> java.lang.AssertionError: expected:<27241> but was:<27245>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ipc.TestCallQueueManager.testSwapUnderContention(TestCallQueueManager.java:220)
> {code}
> It looked like the elements in the queue are dropped due to
> {{CallQueueManager#swapQueue}}
> Looked at the implementation of {{CallQueueManager#swapQueue}}, there is a
> possibility that the elements in the queue are dropped. If the queue is full,
> the calling thread for {{CallQueueManager#put}} is blocked for long time. It
> may put the element into the old queue after queue in {{takeRef}} is changed
> by swapQueue, then this element in the old queue will be dropped.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12189) improve CallQueueManager#swapQueue to
[
https://issues.apache.org/jira/browse/HADOOP-12189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated HADOOP-12189:
---
Issue Type: Improvement (was: Bug)
> improve CallQueueManager#swapQueue to
> --
>
> Key: HADOOP-12189
> URL: https://issues.apache.org/jira/browse/HADOOP-12189
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, test
>Affects Versions: 2.7.1
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: HADOOP-12189.000.patch, HADOOP-12189.001.patch,
> HADOOP-12189.none_guarantee.000.patch, HADOOP-12189.none_guarantee.001.patch,
> HADOOP-12189.none_guarantee.002.patch
>
>
> CallQueueManager may drop elements from the queue sometimes when calling
> {{swapQueue}}.
> The following test failure from TestCallQueueManager shown some elements in
> the queue are dropped.
> https://builds.apache.org/job/PreCommit-HADOOP-Build/7150/testReport/org.apache.hadoop.ipc/TestCallQueueManager/testSwapUnderContention/
> {code}
> java.lang.AssertionError: expected:<27241> but was:<27245>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ipc.TestCallQueueManager.testSwapUnderContention(TestCallQueueManager.java:220)
> {code}
> It looked like the elements in the queue are dropped due to
> {{CallQueueManager#swapQueue}}
> Looked at the implementation of {{CallQueueManager#swapQueue}}, there is a
> possibility that the elements in the queue are dropped. If the queue is full,
> the calling thread for {{CallQueueManager#put}} is blocked for long time. It
> may put the element into the old queue after queue in {{takeRef}} is changed
> by swapQueue, then this element in the old queue will be dropped.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12189) improve CallQueueManager#swapQueue to make queue elements drop nearly impossible.
[
https://issues.apache.org/jira/browse/HADOOP-12189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated HADOOP-12189:
---
Summary: improve CallQueueManager#swapQueue to make queue elements drop
nearly impossible. (was: improve CallQueueManager#swapQueue to )
> improve CallQueueManager#swapQueue to make queue elements drop nearly
> impossible.
> -
>
> Key: HADOOP-12189
> URL: https://issues.apache.org/jira/browse/HADOOP-12189
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, test
>Affects Versions: 2.7.1
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: HADOOP-12189.000.patch, HADOOP-12189.001.patch,
> HADOOP-12189.none_guarantee.000.patch, HADOOP-12189.none_guarantee.001.patch,
> HADOOP-12189.none_guarantee.002.patch
>
>
> CallQueueManager may drop elements from the queue sometimes when calling
> {{swapQueue}}.
> The following test failure from TestCallQueueManager shown some elements in
> the queue are dropped.
> https://builds.apache.org/job/PreCommit-HADOOP-Build/7150/testReport/org.apache.hadoop.ipc/TestCallQueueManager/testSwapUnderContention/
> {code}
> java.lang.AssertionError: expected:<27241> but was:<27245>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ipc.TestCallQueueManager.testSwapUnderContention(TestCallQueueManager.java:220)
> {code}
> It looked like the elements in the queue are dropped due to
> {{CallQueueManager#swapQueue}}
> Looked at the implementation of {{CallQueueManager#swapQueue}}, there is a
> possibility that the elements in the queue are dropped. If the queue is full,
> the calling thread for {{CallQueueManager#put}} is blocked for long time. It
> may put the element into the old queue after queue in {{takeRef}} is changed
> by swapQueue, then this element in the old queue will be dropped.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12189) improve CallQueueManager#swapQueue to make queue elements drop nearly impossible.
[
https://issues.apache.org/jira/browse/HADOOP-12189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639544#comment-14639544
]
Andrew Wang commented on HADOOP-12189:
--
LGTM too, will commit shortly. Thanks Zhihai for the patch, Chris and Arpit for
reviewing.
> improve CallQueueManager#swapQueue to make queue elements drop nearly
> impossible.
> -
>
> Key: HADOOP-12189
> URL: https://issues.apache.org/jira/browse/HADOOP-12189
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, test
>Affects Versions: 2.7.1
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: HADOOP-12189.000.patch, HADOOP-12189.001.patch,
> HADOOP-12189.none_guarantee.000.patch, HADOOP-12189.none_guarantee.001.patch,
> HADOOP-12189.none_guarantee.002.patch
>
>
> CallQueueManager may drop elements from the queue sometimes when calling
> {{swapQueue}}.
> The following test failure from TestCallQueueManager shown some elements in
> the queue are dropped.
> https://builds.apache.org/job/PreCommit-HADOOP-Build/7150/testReport/org.apache.hadoop.ipc/TestCallQueueManager/testSwapUnderContention/
> {code}
> java.lang.AssertionError: expected:<27241> but was:<27245>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ipc.TestCallQueueManager.testSwapUnderContention(TestCallQueueManager.java:220)
> {code}
> It looked like the elements in the queue are dropped due to
> {{CallQueueManager#swapQueue}}
> Looked at the implementation of {{CallQueueManager#swapQueue}}, there is a
> possibility that the elements in the queue are dropped. If the queue is full,
> the calling thread for {{CallQueueManager#put}} is blocked for long time. It
> may put the element into the old queue after queue in {{takeRef}} is changed
> by swapQueue, then this element in the old queue will be dropped.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12189) Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible.
[
https://issues.apache.org/jira/browse/HADOOP-12189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated HADOOP-12189:
---
Description:
Improve CallQueueManager#swapQueue to make queue elements drop nearly
impossible. This is the trade-off between performance and functionality, even
in the really really rare situation, we may drop one element, but it is not the
end of the world since the client may still recover with timeout.
CallQueueManager may drop elements from the queue sometimes when calling
{{swapQueue}}.
The following test failure from TestCallQueueManager shown some elements in the
queue are dropped.
https://builds.apache.org/job/PreCommit-HADOOP-Build/7150/testReport/org.apache.hadoop.ipc/TestCallQueueManager/testSwapUnderContention/
{code}
java.lang.AssertionError: expected:<27241> but was:<27245>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.ipc.TestCallQueueManager.testSwapUnderContention(TestCallQueueManager.java:220)
{code}
It looked like the elements in the queue are dropped due to
{{CallQueueManager#swapQueue}}
Looked at the implementation of {{CallQueueManager#swapQueue}}, there is a
possibility that the elements in the queue are dropped. If the queue is full,
the calling thread for {{CallQueueManager#put}} is blocked for long time. It
may put the element into the old queue after queue in {{takeRef}} is changed by
swapQueue, then this element in the old queue will be dropped.
was:
CallQueueManager may drop elements from the queue sometimes when calling
{{swapQueue}}.
The following test failure from TestCallQueueManager shown some elements in the
queue are dropped.
https://builds.apache.org/job/PreCommit-HADOOP-Build/7150/testReport/org.apache.hadoop.ipc/TestCallQueueManager/testSwapUnderContention/
{code}
java.lang.AssertionError: expected:<27241> but was:<27245>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.ipc.TestCallQueueManager.testSwapUnderContention(TestCallQueueManager.java:220)
{code}
It looked like the elements in the queue are dropped due to
{{CallQueueManager#swapQueue}}
Looked at the implementation of {{CallQueueManager#swapQueue}}, there is a
possibility that the elements in the queue are dropped. If the queue is full,
the calling thread for {{CallQueueManager#put}} is blocked for long time. It
may put the element into the old queue after queue in {{takeRef}} is changed by
swapQueue, then this element in the old queue will be dropped.
Summary: Improve CallQueueManager#swapQueue to make queue elements drop
nearly impossible. (was: improve CallQueueManager#swapQueue to make queue
elements drop nearly impossible.)
> Improve CallQueueManager#swapQueue to make queue elements drop nearly
> impossible.
> -
>
> Key: HADOOP-12189
> URL: https://issues.apache.org/jira/browse/HADOOP-12189
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, test
>Affects Versions: 2.7.1
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: HADOOP-12189.000.patch, HADOOP-12189.001.patch,
> HADOOP-12189.none_guarantee.000.patch, HADOOP-12189.none_guarantee.001.patch,
> HADOOP-12189.none_guarantee.002.patch
>
>
> Improve CallQueueManager#swapQueue to make queue elements drop nearly
> impossible. This is the trade-off between performance and functionality, even
> in the really really rare situation, we may drop one element, but it is not
> the end of the world since the client may still recover with timeout.
> CallQueueManager may drop elements from the queue sometimes when calling
> {{swapQueue}}.
> The following test failure from TestCallQueueManager shown some elements in
> the queue are dropped.
> https://builds.apache.org/job/PreCommit-HADOOP-Build/7150/testReport/org.apache.hadoop.ipc/TestCallQueueManager/testSwapUnderContention/
> {code}
> java.lang.AssertionError: expected:<27241> but was:<27245>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ipc.TestCallQueueManager.testSwapUnderContention(TestCallQue
[jira] [Updated] (HADOOP-12189) Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible.
[
https://issues.apache.org/jira/browse/HADOOP-12189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated HADOOP-12189:
---
Description:
Improve CallQueueManager#swapQueue to make queue elements drop nearly
impossible. This is the trade-off between performance and functionality, even
in the very very rare situation, we may drop one element, but it is not the end
of the world since the client may still recover with timeout.
CallQueueManager may drop elements from the queue sometimes when calling
{{swapQueue}}.
The following test failure from TestCallQueueManager shown some elements in the
queue are dropped.
https://builds.apache.org/job/PreCommit-HADOOP-Build/7150/testReport/org.apache.hadoop.ipc/TestCallQueueManager/testSwapUnderContention/
{code}
java.lang.AssertionError: expected:<27241> but was:<27245>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.ipc.TestCallQueueManager.testSwapUnderContention(TestCallQueueManager.java:220)
{code}
It looked like the elements in the queue are dropped due to
{{CallQueueManager#swapQueue}}
Looked at the implementation of {{CallQueueManager#swapQueue}}, there is a
possibility that the elements in the queue are dropped. If the queue is full,
the calling thread for {{CallQueueManager#put}} is blocked for long time. It
may put the element into the old queue after queue in {{takeRef}} is changed by
swapQueue, then this element in the old queue will be dropped.
was:
Improve CallQueueManager#swapQueue to make queue elements drop nearly
impossible. This is the trade-off between performance and functionality, even
in the really really rare situation, we may drop one element, but it is not the
end of the world since the client may still recover with timeout.
CallQueueManager may drop elements from the queue sometimes when calling
{{swapQueue}}.
The following test failure from TestCallQueueManager shown some elements in the
queue are dropped.
https://builds.apache.org/job/PreCommit-HADOOP-Build/7150/testReport/org.apache.hadoop.ipc/TestCallQueueManager/testSwapUnderContention/
{code}
java.lang.AssertionError: expected:<27241> but was:<27245>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.ipc.TestCallQueueManager.testSwapUnderContention(TestCallQueueManager.java:220)
{code}
It looked like the elements in the queue are dropped due to
{{CallQueueManager#swapQueue}}
Looked at the implementation of {{CallQueueManager#swapQueue}}, there is a
possibility that the elements in the queue are dropped. If the queue is full,
the calling thread for {{CallQueueManager#put}} is blocked for long time. It
may put the element into the old queue after queue in {{takeRef}} is changed by
swapQueue, then this element in the old queue will be dropped.
> Improve CallQueueManager#swapQueue to make queue elements drop nearly
> impossible.
> -
>
> Key: HADOOP-12189
> URL: https://issues.apache.org/jira/browse/HADOOP-12189
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, test
>Affects Versions: 2.7.1
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: HADOOP-12189.000.patch, HADOOP-12189.001.patch,
> HADOOP-12189.none_guarantee.000.patch, HADOOP-12189.none_guarantee.001.patch,
> HADOOP-12189.none_guarantee.002.patch
>
>
> Improve CallQueueManager#swapQueue to make queue elements drop nearly
> impossible. This is the trade-off between performance and functionality, even
> in the very very rare situation, we may drop one element, but it is not the
> end of the world since the client may still recover with timeout.
> CallQueueManager may drop elements from the queue sometimes when calling
> {{swapQueue}}.
> The following test failure from TestCallQueueManager shown some elements in
> the queue are dropped.
> https://builds.apache.org/job/PreCommit-HADOOP-Build/7150/testReport/org.apache.hadoop.ipc/TestCallQueueManager/testSwapUnderContention/
> {code}
> java.lang.AssertionError: expected:<27241> but was:<27245>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.jav
[jira] [Updated] (HADOOP-12189) Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible.
[
https://issues.apache.org/jira/browse/HADOOP-12189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-12189:
-
Resolution: Fixed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
I committed this to branch-2 and trunk, since it seems like essentially a test
fix I don't think it needs to go to branch-2.7. Thanks again all.
> Improve CallQueueManager#swapQueue to make queue elements drop nearly
> impossible.
> -
>
> Key: HADOOP-12189
> URL: https://issues.apache.org/jira/browse/HADOOP-12189
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, test
>Affects Versions: 2.7.1
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.8.0
>
> Attachments: HADOOP-12189.000.patch, HADOOP-12189.001.patch,
> HADOOP-12189.none_guarantee.000.patch, HADOOP-12189.none_guarantee.001.patch,
> HADOOP-12189.none_guarantee.002.patch
>
>
> Improve CallQueueManager#swapQueue to make queue elements drop nearly
> impossible. This is the trade-off between performance and functionality, even
> in the very very rare situation, we may drop one element, but it is not the
> end of the world since the client may still recover with timeout.
> CallQueueManager may drop elements from the queue sometimes when calling
> {{swapQueue}}.
> The following test failure from TestCallQueueManager shown some elements in
> the queue are dropped.
> https://builds.apache.org/job/PreCommit-HADOOP-Build/7150/testReport/org.apache.hadoop.ipc/TestCallQueueManager/testSwapUnderContention/
> {code}
> java.lang.AssertionError: expected:<27241> but was:<27245>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ipc.TestCallQueueManager.testSwapUnderContention(TestCallQueueManager.java:220)
> {code}
> It looked like the elements in the queue are dropped due to
> {{CallQueueManager#swapQueue}}
> Looked at the implementation of {{CallQueueManager#swapQueue}}, there is a
> possibility that the elements in the queue are dropped. If the queue is full,
> the calling thread for {{CallQueueManager#put}} is blocked for long time. It
> may put the element into the old queue after queue in {{takeRef}} is changed
> by swapQueue, then this element in the old queue will be dropped.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-12264) Update create-release.sh to pass -Preleasedocs
[ https://issues.apache.org/jira/browse/HADOOP-12264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-12264. -- Resolution: Duplicate This is a dupe of HADOOP-11793 > Update create-release.sh to pass -Preleasedocs > -- > > Key: HADOOP-12264 > URL: https://issues.apache.org/jira/browse/HADOOP-12264 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > > We use the create-release.sh script to create the release artifacts. With the > way of autogenerating CHANGES.txt we need to update the script to also pass > "-Preleasedocs". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12189) Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible.
[
https://issues.apache.org/jira/browse/HADOOP-12189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639568#comment-14639568
]
zhihai xu commented on HADOOP-12189:
Thanks Chris and Arpit for reviewing! thanks Andrew for reviewing and
committing the patch!
> Improve CallQueueManager#swapQueue to make queue elements drop nearly
> impossible.
> -
>
> Key: HADOOP-12189
> URL: https://issues.apache.org/jira/browse/HADOOP-12189
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, test
>Affects Versions: 2.7.1
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.8.0
>
> Attachments: HADOOP-12189.000.patch, HADOOP-12189.001.patch,
> HADOOP-12189.none_guarantee.000.patch, HADOOP-12189.none_guarantee.001.patch,
> HADOOP-12189.none_guarantee.002.patch
>
>
> Improve CallQueueManager#swapQueue to make queue elements drop nearly
> impossible. This is the trade-off between performance and functionality, even
> in the very very rare situation, we may drop one element, but it is not the
> end of the world since the client may still recover with timeout.
> CallQueueManager may drop elements from the queue sometimes when calling
> {{swapQueue}}.
> The following test failure from TestCallQueueManager shown some elements in
> the queue are dropped.
> https://builds.apache.org/job/PreCommit-HADOOP-Build/7150/testReport/org.apache.hadoop.ipc/TestCallQueueManager/testSwapUnderContention/
> {code}
> java.lang.AssertionError: expected:<27241> but was:<27245>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ipc.TestCallQueueManager.testSwapUnderContention(TestCallQueueManager.java:220)
> {code}
> It looked like the elements in the queue are dropped due to
> {{CallQueueManager#swapQueue}}
> Looked at the implementation of {{CallQueueManager#swapQueue}}, there is a
> possibility that the elements in the queue are dropped. If the queue is full,
> the calling thread for {{CallQueueManager#put}} is blocked for long time. It
> may put the element into the old queue after queue in {{takeRef}} is changed
> by swapQueue, then this element in the old queue will be dropped.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11793) Update create-release for releasedocmaker.py
[ https://issues.apache.org/jira/browse/HADOOP-11793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-11793: - Affects Version/s: (was: 3.0.0) 2.8.0 Target Version/s: 2.8.0 > Update create-release for releasedocmaker.py > > > Key: HADOOP-11793 > URL: https://issues.apache.org/jira/browse/HADOOP-11793 > Project: Hadoop Common > Issue Type: Task > Components: build >Affects Versions: 2.8.0 >Reporter: Allen Wittenauer >Assignee: ramtin > Attachments: HADOOP-11793.001.patch > > > With the commit of HADOOP-11731, the changelog and release note data is now > automated with the build. The create-release script needs to do the correct > thing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11793) Update create-release for releasedocmaker.py
[
https://issues.apache.org/jira/browse/HADOOP-11793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639570#comment-14639570
]
Andrew Wang commented on HADOOP-11793:
--
This looks good to me, though I think we could skip the second {{mvn clean}}
invocation. [~aw] any thoughts?
> Update create-release for releasedocmaker.py
>
>
> Key: HADOOP-11793
> URL: https://issues.apache.org/jira/browse/HADOOP-11793
> Project: Hadoop Common
> Issue Type: Task
> Components: build
>Affects Versions: 2.8.0
>Reporter: Allen Wittenauer
>Assignee: ramtin
> Attachments: HADOOP-11793.001.patch
>
>
> With the commit of HADOOP-11731, the changelog and release note data is now
> automated with the build. The create-release script needs to do the correct
> thing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12189) Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible.
[
https://issues.apache.org/jira/browse/HADOOP-12189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639579#comment-14639579
]
Hudson commented on HADOOP-12189:
-
FAILURE: Integrated in Hadoop-trunk-Commit #8208 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8208/])
HADOOP-12189. Improve CallQueueManager#swapQueue to make queue elements drop
nearly impossible. Contributed by Zhihai Xu. (wang: rev
6736a1ab7033523ed5f304fdfed46d7f348665b4)
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java
* hadoop-common-project/hadoop-common/CHANGES.txt
> Improve CallQueueManager#swapQueue to make queue elements drop nearly
> impossible.
> -
>
> Key: HADOOP-12189
> URL: https://issues.apache.org/jira/browse/HADOOP-12189
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, test
>Affects Versions: 2.7.1
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.8.0
>
> Attachments: HADOOP-12189.000.patch, HADOOP-12189.001.patch,
> HADOOP-12189.none_guarantee.000.patch, HADOOP-12189.none_guarantee.001.patch,
> HADOOP-12189.none_guarantee.002.patch
>
>
> Improve CallQueueManager#swapQueue to make queue elements drop nearly
> impossible. This is the trade-off between performance and functionality, even
> in the very very rare situation, we may drop one element, but it is not the
> end of the world since the client may still recover with timeout.
> CallQueueManager may drop elements from the queue sometimes when calling
> {{swapQueue}}.
> The following test failure from TestCallQueueManager shown some elements in
> the queue are dropped.
> https://builds.apache.org/job/PreCommit-HADOOP-Build/7150/testReport/org.apache.hadoop.ipc/TestCallQueueManager/testSwapUnderContention/
> {code}
> java.lang.AssertionError: expected:<27241> but was:<27245>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ipc.TestCallQueueManager.testSwapUnderContention(TestCallQueueManager.java:220)
> {code}
> It looked like the elements in the queue are dropped due to
> {{CallQueueManager#swapQueue}}
> Looked at the implementation of {{CallQueueManager#swapQueue}}, there is a
> possibility that the elements in the queue are dropped. If the queue is full,
> the calling thread for {{CallQueueManager#put}} is blocked for long time. It
> may put the element into the old queue after queue in {{takeRef}} is changed
> by swapQueue, then this element in the old queue will be dropped.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11793) Update create-release for releasedocmaker.py
[ https://issues.apache.org/jira/browse/HADOOP-11793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639701#comment-14639701 ] Allen Wittenauer commented on HADOOP-11793: --- The reason why the lint mode was added so that create-release would purposefully stop the build so that the RM could fix issues in JIRA prior to cutting the release. So create-release ideally would have this change *and* another one to independently call releasedocmaker in lint mode very early on. Additionally, the wiki page on how to actually use create release needs to be majorly updated with steps re-arranged, otherwise Bad Things(tm) will happen. In particular, closing the release out in jira means those issues cannot be updated. (e.g., YARN-3614 will probably never show up in releasedocmaker output because it was Fixed incorrectly and then subsequently closed by the RM.) > Update create-release for releasedocmaker.py > > > Key: HADOOP-11793 > URL: https://issues.apache.org/jira/browse/HADOOP-11793 > Project: Hadoop Common > Issue Type: Task > Components: build >Affects Versions: 2.8.0 >Reporter: Allen Wittenauer >Assignee: ramtin > Attachments: HADOOP-11793.001.patch > > > With the commit of HADOOP-11731, the changelog and release note data is now > automated with the build. The create-release script needs to do the correct > thing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11791) Update src/site/markdown/releases to include old versions of Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-11791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639703#comment-14639703 ] Allen Wittenauer commented on HADOOP-11791: --- Yetus has the ability to do this much easier now. Should probably just updated create-release to do this at release time. > Update src/site/markdown/releases to include old versions of Hadoop > --- > > Key: HADOOP-11791 > URL: https://issues.apache.org/jira/browse/HADOOP-11791 > Project: Hadoop Common > Issue Type: Task > Components: build, documentation >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer > Attachments: HADOOP-11791.001.patch > > > With the commit of HADOOP-11731, we need to include the new format of release > information in trunk. This JIRA is about including those old versions in the > tree. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HADOOP-10854) unit tests for the shell scripts
[
https://issues.apache.org/jira/browse/HADOOP-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639708#comment-14639708
]
Allen Wittenauer edited comment on HADOOP-10854 at 7/24/15 12:14 AM:
-
Yeah, basically:
{{mvn test -Pshelltest -DskipTests}} will only run the shelltest.
{{mvn test -Pshelltest}} will only run the unit tests and not both.
I've got an updated version of this patch that:
a) won't fail maven if bats isn't installed
b) moves the phase for shelltest to process-test-classes. This gets run after
test-compile and before test. This fixes the {[mvn test -Pshelltest}} case,
but it's still less than ideal.
was (Author: aw):
Yeah, basically:
{{mvn test -Pshelltest -DskipTests}} will only run the shelltest.
{[mvn test -Pshelltest}} will only run the unit tests and not both.
I've got an updated version of this patch that:
a) won't fail maven if bats isn't installed
b) moves the phase for shelltest to process-test-classes. This gets run after
test-compile and before test. This fixes the {[mvn test -Pshelltest}} case,
but it's still less than ideal.
> unit tests for the shell scripts
>
>
> Key: HADOOP-10854
> URL: https://issues.apache.org/jira/browse/HADOOP-10854
> Project: Hadoop Common
> Issue Type: Test
> Components: scripts
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
> Labels: scripts
> Attachments: HADOOP-10854.00.patch, HADOOP-10854.01.patch,
> HADOOP-10854.02.patch, HADOOP-10854.03.patch
>
>
> With HADOOP-9902 moving a lot of the core functionality to functions, we
> should build some unit tests for them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-10854) unit tests for the shell scripts
[
https://issues.apache.org/jira/browse/HADOOP-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639708#comment-14639708
]
Allen Wittenauer commented on HADOOP-10854:
---
Yeah, basically:
{{mvn test -Pshelltest -DskipTests}} will only run the shelltest.
{[mvn test -Pshelltest}} will only run the unit tests and not both.
I've got an updated version of this patch that:
a) won't fail maven if bats isn't installed
b) moves the phase for shelltest to process-test-classes. This gets run after
test-compile and before test. This fixes the {[mvn test -Pshelltest}} case,
but it's still less than ideal.
> unit tests for the shell scripts
>
>
> Key: HADOOP-10854
> URL: https://issues.apache.org/jira/browse/HADOOP-10854
> Project: Hadoop Common
> Issue Type: Test
> Components: scripts
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
> Labels: scripts
> Attachments: HADOOP-10854.00.patch, HADOOP-10854.01.patch,
> HADOOP-10854.02.patch, HADOOP-10854.03.patch
>
>
> With HADOOP-9902 moving a lot of the core functionality to functions, we
> should build some unit tests for them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HADOOP-10854) unit tests for the shell scripts
[
https://issues.apache.org/jira/browse/HADOOP-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639708#comment-14639708
]
Allen Wittenauer edited comment on HADOOP-10854 at 7/24/15 12:14 AM:
-
Yeah, basically:
{{mvn test -Pshelltest -DskipTests}} will only run the shelltest.
{{mvn test -Pshelltest}} will only run the unit tests and not both.
I've got an updated version of this patch that:
a) won't fail maven if bats isn't installed
b) moves the phase for shelltest to process-test-classes. This gets run after
test-compile and before test. This fixes the {{mvn test -Pshelltest}} case,
but it's still less than ideal.
was (Author: aw):
Yeah, basically:
{{mvn test -Pshelltest -DskipTests}} will only run the shelltest.
{{mvn test -Pshelltest}} will only run the unit tests and not both.
I've got an updated version of this patch that:
a) won't fail maven if bats isn't installed
b) moves the phase for shelltest to process-test-classes. This gets run after
test-compile and before test. This fixes the {[mvn test -Pshelltest}} case,
but it's still less than ideal.
> unit tests for the shell scripts
>
>
> Key: HADOOP-10854
> URL: https://issues.apache.org/jira/browse/HADOOP-10854
> Project: Hadoop Common
> Issue Type: Test
> Components: scripts
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
> Labels: scripts
> Attachments: HADOOP-10854.00.patch, HADOOP-10854.01.patch,
> HADOOP-10854.02.patch, HADOOP-10854.03.patch
>
>
> With HADOOP-9902 moving a lot of the core functionality to functions, we
> should build some unit tests for them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12226) CHANGED_MODULES is wrong for ant
[ https://issues.apache.org/jira/browse/HADOOP-12226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639712#comment-14639712 ] Allen Wittenauer commented on HADOOP-12226: --- There's a (minor) bug in this code: QETESTMODE is never set to false so CHANGED_MODULES is blank. I'll post an addendum to fix it. > CHANGED_MODULES is wrong for ant > > > Key: HADOOP-12226 > URL: https://issues.apache.org/jira/browse/HADOOP-12226 > Project: Hadoop Common > Issue Type: Sub-task > Components: yetus >Affects Versions: HADOOP-12111 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Fix For: HADOOP-12111 > > Attachments: HADOOP-12226.HADOOP-12111.00.patch > > > CHANGED_MODULES assumes maven and will have incorrect results for ant. > We may need to rethink how this and CHANGED_UNFILTERED_MODULES work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HADOOP-12226) CHANGED_MODULES is wrong for ant
[ https://issues.apache.org/jira/browse/HADOOP-12226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer reopened HADOOP-12226: --- > CHANGED_MODULES is wrong for ant > > > Key: HADOOP-12226 > URL: https://issues.apache.org/jira/browse/HADOOP-12226 > Project: Hadoop Common > Issue Type: Sub-task > Components: yetus >Affects Versions: HADOOP-12111 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Fix For: HADOOP-12111 > > Attachments: HADOOP-12226.HADOOP-12111.00.patch > > > CHANGED_MODULES assumes maven and will have incorrect results for ant. > We may need to rethink how this and CHANGED_UNFILTERED_MODULES work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12226) CHANGED_MODULES is wrong for ant
[ https://issues.apache.org/jira/browse/HADOOP-12226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-12226: -- Attachment: HADOOP-12226-addendum.00.patch addendum.00: * remove the unit test code that snuck in > CHANGED_MODULES is wrong for ant > > > Key: HADOOP-12226 > URL: https://issues.apache.org/jira/browse/HADOOP-12226 > Project: Hadoop Common > Issue Type: Sub-task > Components: yetus >Affects Versions: HADOOP-12111 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Fix For: HADOOP-12111 > > Attachments: HADOOP-12226-addendum.00.patch, > HADOOP-12226.HADOOP-12111.00.patch > > > CHANGED_MODULES assumes maven and will have incorrect results for ant. > We may need to rethink how this and CHANGED_UNFILTERED_MODULES work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12226) CHANGED_MODULES is wrong for ant
[ https://issues.apache.org/jira/browse/HADOOP-12226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-12226: -- Status: Patch Available (was: Reopened) > CHANGED_MODULES is wrong for ant > > > Key: HADOOP-12226 > URL: https://issues.apache.org/jira/browse/HADOOP-12226 > Project: Hadoop Common > Issue Type: Sub-task > Components: yetus >Affects Versions: HADOOP-12111 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Fix For: HADOOP-12111 > > Attachments: HADOOP-12226-addendum.00.patch, > HADOOP-12226.HADOOP-12111.00.patch > > > CHANGED_MODULES assumes maven and will have incorrect results for ant. > We may need to rethink how this and CHANGED_UNFILTERED_MODULES work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12009) Clarify FileSystem.listStatus() sorting order & fix FileSystemContractBaseTest:testListStatus
[
https://issues.apache.org/jira/browse/HADOOP-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639742#comment-14639742
]
Jakob Homan commented on HADOOP-12009:
--
I've verified that the latest patch (v4) fixes the observed test failures
(munged the test to verify the contract implementations were getting the latest
versions and still passing). I'm going to go ahead and commit.
bq. If the patch is causing problems, I suggest that before it gets committed
the full contract test suite —including the s3, openstack and azure suites— get
run before we put this in.
Are there instructions for that? Are any vendors providing that service?
> Clarify FileSystem.listStatus() sorting order & fix
> FileSystemContractBaseTest:testListStatus
> --
>
> Key: HADOOP-12009
> URL: https://issues.apache.org/jira/browse/HADOOP-12009
> Project: Hadoop Common
> Issue Type: Improvement
> Components: test
>Reporter: Jakob Homan
>Assignee: J.Andreina
>Priority: Minor
> Attachments: HADOOP-12009-003.patch, HADOOP-12009-004.patch,
> HADOOP-12009.1.patch
>
>
> FileSystem.listStatus does not guarantee that implementations will return
> sorted entries:
> {code} /**
>* List the statuses of the files/directories in the given path if the path
> is
>* a directory.
>*
>* @param f given path
>* @return the statuses of the files/directories in the given patch
>* @throws FileNotFoundException when the path does not exist;
>* IOException see specific implementation
>*/
> public abstract FileStatus[] listStatus(Path f) throws
> FileNotFoundException,
> IOException;{code}
> However, FileSystemContractBaseTest, expects the elements to come back sorted:
> {code}Path[] testDirs = { path("/test/hadoop/a"),
> path("/test/hadoop/b"),
> path("/test/hadoop/c/1"), };
>
> // ...
> paths = fs.listStatus(path("/test/hadoop"));
> assertEquals(3, paths.length);
> assertEquals(path("/test/hadoop/a"), paths[0].getPath());
> assertEquals(path("/test/hadoop/b"), paths[1].getPath());
> assertEquals(path("/test/hadoop/c"), paths[2].getPath());{code}
> We should pass this test as long as all the paths are there, regardless of
> their ordering.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12009) Clarify FileSystem.listStatus() sorting order & fix FileSystemContractBaseTest:testListStatus
[
https://issues.apache.org/jira/browse/HADOOP-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated HADOOP-12009:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
I've committed this. Resolving. Thanks, J.!
> Clarify FileSystem.listStatus() sorting order & fix
> FileSystemContractBaseTest:testListStatus
> --
>
> Key: HADOOP-12009
> URL: https://issues.apache.org/jira/browse/HADOOP-12009
> Project: Hadoop Common
> Issue Type: Improvement
> Components: test
>Reporter: Jakob Homan
>Assignee: J.Andreina
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HADOOP-12009-003.patch, HADOOP-12009-004.patch,
> HADOOP-12009.1.patch
>
>
> FileSystem.listStatus does not guarantee that implementations will return
> sorted entries:
> {code} /**
>* List the statuses of the files/directories in the given path if the path
> is
>* a directory.
>*
>* @param f given path
>* @return the statuses of the files/directories in the given patch
>* @throws FileNotFoundException when the path does not exist;
>* IOException see specific implementation
>*/
> public abstract FileStatus[] listStatus(Path f) throws
> FileNotFoundException,
> IOException;{code}
> However, FileSystemContractBaseTest, expects the elements to come back sorted:
> {code}Path[] testDirs = { path("/test/hadoop/a"),
> path("/test/hadoop/b"),
> path("/test/hadoop/c/1"), };
>
> // ...
> paths = fs.listStatus(path("/test/hadoop"));
> assertEquals(3, paths.length);
> assertEquals(path("/test/hadoop/a"), paths[0].getPath());
> assertEquals(path("/test/hadoop/b"), paths[1].getPath());
> assertEquals(path("/test/hadoop/c"), paths[2].getPath());{code}
> We should pass this test as long as all the paths are there, regardless of
> their ordering.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12265) Pylint should be installed in test-patch docker environment
Kengo Seki created HADOOP-12265: --- Summary: Pylint should be installed in test-patch docker environment Key: HADOOP-12265 URL: https://issues.apache.org/jira/browse/HADOOP-12265 Project: Hadoop Common Issue Type: Sub-task Components: yetus Reporter: Kengo Seki HADOOP-12207 added pylint plugin to test-patch, but pylint won't be installed in Docker environment because I forgot modify Dockerfile :) It must be updated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12265) Pylint should be installed in test-patch docker environment
[ https://issues.apache.org/jira/browse/HADOOP-12265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639777#comment-14639777 ] Kengo Seki commented on HADOOP-12265: - Documents should be updated also. > Pylint should be installed in test-patch docker environment > --- > > Key: HADOOP-12265 > URL: https://issues.apache.org/jira/browse/HADOOP-12265 > Project: Hadoop Common > Issue Type: Sub-task > Components: yetus >Reporter: Kengo Seki > > HADOOP-12207 added pylint plugin to test-patch, but pylint won't be installed > in Docker environment because I forgot modify Dockerfile :) It must be > updated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12266) make test-patch plugins more consistent
Kengo Seki created HADOOP-12266:
---
Summary: make test-patch plugins more consistent
Key: HADOOP-12266
URL: https://issues.apache.org/jira/browse/HADOOP-12266
Project: Hadoop Common
Issue Type: Sub-task
Components: yetus
Reporter: Kengo Seki
In response to:
https://issues.apache.org/jira/browse/HADOOP-12207?focusedCommentId=14637515&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14637515
* make *_calcdiffs to be a common function
* move add_footer_table after all examinations finished
* replace ${PATCH_BRANCH} in output filenames with "branch-"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12009) Clarify FileSystem.listStatus() sorting order & fix FileSystemContractBaseTest:testListStatus
[
https://issues.apache.org/jira/browse/HADOOP-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639828#comment-14639828
]
Hudson commented on HADOOP-12009:
-
SUCCESS: Integrated in Hadoop-trunk-Commit #8209 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8209/])
HADOOP-12009: Clarify FileSystem.listStatus() sorting order & fix
FileSystemContractBaseTest:testListStatus. (J.Andreina via jghoman) (jghoman:
rev ab3197c20452e0dd908193d6854c204e6ee34645)
* hadoop-common-project/hadoop-common/CHANGES.txt
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
* hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
> Clarify FileSystem.listStatus() sorting order & fix
> FileSystemContractBaseTest:testListStatus
> --
>
> Key: HADOOP-12009
> URL: https://issues.apache.org/jira/browse/HADOOP-12009
> Project: Hadoop Common
> Issue Type: Improvement
> Components: test
>Reporter: Jakob Homan
>Assignee: J.Andreina
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HADOOP-12009-003.patch, HADOOP-12009-004.patch,
> HADOOP-12009.1.patch
>
>
> FileSystem.listStatus does not guarantee that implementations will return
> sorted entries:
> {code} /**
>* List the statuses of the files/directories in the given path if the path
> is
>* a directory.
>*
>* @param f given path
>* @return the statuses of the files/directories in the given patch
>* @throws FileNotFoundException when the path does not exist;
>* IOException see specific implementation
>*/
> public abstract FileStatus[] listStatus(Path f) throws
> FileNotFoundException,
> IOException;{code}
> However, FileSystemContractBaseTest, expects the elements to come back sorted:
> {code}Path[] testDirs = { path("/test/hadoop/a"),
> path("/test/hadoop/b"),
> path("/test/hadoop/c/1"), };
>
> // ...
> paths = fs.listStatus(path("/test/hadoop"));
> assertEquals(3, paths.length);
> assertEquals(path("/test/hadoop/a"), paths[0].getPath());
> assertEquals(path("/test/hadoop/b"), paths[1].getPath());
> assertEquals(path("/test/hadoop/c"), paths[2].getPath());{code}
> We should pass this test as long as all the paths are there, regardless of
> their ordering.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12259) Utility to Dynamic port allocation
[ https://issues.apache.org/jira/browse/HADOOP-12259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639884#comment-14639884 ] Brahma Reddy Battula commented on HADOOP-12259: --- thanks a lot [~rkanter] and [~varun_saxena] for your reviews. > Utility to Dynamic port allocation > -- > > Key: HADOOP-12259 > URL: https://issues.apache.org/jira/browse/HADOOP-12259 > Project: Hadoop Common > Issue Type: Bug > Components: test, util >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HADOOP-12259.patch > > > As per discussion in YARN-3528 and [~rkanter] comment [here | > https://issues.apache.org/jira/browse/YARN-3528?focusedCommentId=14637700&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14637700 > ] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12267) s3a failure due to integer overflow bug in AWS SDK
Aaron Fabbri created HADOOP-12267: - Summary: s3a failure due to integer overflow bug in AWS SDK Key: HADOOP-12267 URL: https://issues.apache.org/jira/browse/HADOOP-12267 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 2.6.0 Reporter: Aaron Fabbri Assignee: Aaron Fabbri Under high load writing to Amazon AWS S3 storage, a client can be throttled enough to encounter 24 retries in a row. The amazon http client code (in aws-java-sdk jar) has a bug in its exponential backoff retry code, that causes integer overflow, and a call to Thread.sleep() with a negative value, which causes client to bail out with an exception (see below). Bug has been fixed in aws-java-sdk: https://github.com/aws/aws-sdk-java/pull/388 We need to pick this up for hadoop-tools/hadoop-aws. Error: java.io.IOException: File copy failed: hdfs://path-redacted --> s3a://path-redacted at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:284) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:252) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hdfs://path-redacted to s3a://path-redacted at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:280) ... 10 more Caused by: com.amazonaws.AmazonClientException: Unable to complete transfer: timeout value is negative at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:300) at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:284) at com.amazonaws.services.s3.transfer.internal.CopyImpl.waitForCopyResult(CopyImpl.java:67) at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:943) at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:357) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:137) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:100) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Caused by: java.lang.IllegalArgumentException: timeout value is negative at java.lang.Thread.sleep(Native Method) at com.amazonaws.http.AmazonHttpClient.pauseBeforeNextRetry(AmazonHttpClient.java:864) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:353) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) at com.amazonaws.services.s3.AmazonS3Client.copyObject(AmazonS3Client.java:1507) at com.amazonaws.services.s3.transfer.internal.CopyCallable.copyInOneChunk(CopyCallable.java:143) at com.amazonaws.services.s3.transfer.internal.CopyCallable.call(CopyCallable.java:131) at com.amazonaws.services.s3.transfer.internal.CopyMonitor.copy(CopyMonitor.java:189) at com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:134) at com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12267) s3a failure due to integer overflow bug in AWS SDK
[ https://issues.apache.org/jira/browse/HADOOP-12267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-12267: -- Attachment: HADOOP-12267.2.6.0.001-patch > s3a failure due to integer overflow bug in AWS SDK > -- > > Key: HADOOP-12267 > URL: https://issues.apache.org/jira/browse/HADOOP-12267 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.6.0 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri > Attachments: HADOOP-12267.2.6.0.001-patch > > > Under high load writing to Amazon AWS S3 storage, a client can be throttled > enough to encounter 24 retries in a row. > The amazon http client code (in aws-java-sdk jar) has a bug in its > exponential backoff retry code, that causes integer overflow, and a call to > Thread.sleep() with a negative value, which causes client to bail out with an > exception (see below). > Bug has been fixed in aws-java-sdk: > https://github.com/aws/aws-sdk-java/pull/388 > We need to pick this up for hadoop-tools/hadoop-aws. > Error: java.io.IOException: File copy failed: hdfs://path-redacted --> > s3a://path-redacted > at > org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:284) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:252) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: > java.io.IOException: Couldn't run retriable-command: Copying > hdfs://path-redacted to s3a://path-redacted > at > org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) > > at > org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:280) > > ... 10 more > Caused by: com.amazonaws.AmazonClientException: Unable to complete transfer: > timeout value is negative > at > com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:300) > at > com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:284) > at > com.amazonaws.services.s3.transfer.internal.CopyImpl.waitForCopyResult(CopyImpl.java:67) > > at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:943) > at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:357) > at > org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220) > at > org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:137) > > at > org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:100) > at > org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) > > ... 11 more > Caused by: java.lang.IllegalArgumentException: timeout value is negative > at java.lang.Thread.sleep(Native Method) > at > com.amazonaws.http.AmazonHttpClient.pauseBeforeNextRetry(AmazonHttpClient.java:864) > at > com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:353) > at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) > at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) > at > com.amazonaws.services.s3.AmazonS3Client.copyObject(AmazonS3Client.java:1507) > at > com.amazonaws.services.s3.transfer.internal.CopyCallable.copyInOneChunk(CopyCallable.java:143) > at > com.amazonaws.services.s3.transfer.internal.CopyCallable.call(CopyCallable.java:131) > > at > com.amazonaws.services.s3.transfer.internal.CopyMonitor.copy(CopyMonitor.java:189) > > at > com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:134) > > at > com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:46) > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12267) s3a failure due to integer overflow bug in AWS SDK
[ https://issues.apache.org/jira/browse/HADOOP-12267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-12267: -- Attachment: HADOOP-12267.2.6.0.001.patch HADOOP-12267.2.7.1.001.patch > s3a failure due to integer overflow bug in AWS SDK > -- > > Key: HADOOP-12267 > URL: https://issues.apache.org/jira/browse/HADOOP-12267 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.6.0 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri > Attachments: HADOOP-12267.2.6.0.001.patch, > HADOOP-12267.2.7.1.001.patch > > > Under high load writing to Amazon AWS S3 storage, a client can be throttled > enough to encounter 24 retries in a row. > The amazon http client code (in aws-java-sdk jar) has a bug in its > exponential backoff retry code, that causes integer overflow, and a call to > Thread.sleep() with a negative value, which causes client to bail out with an > exception (see below). > Bug has been fixed in aws-java-sdk: > https://github.com/aws/aws-sdk-java/pull/388 > We need to pick this up for hadoop-tools/hadoop-aws. > Error: java.io.IOException: File copy failed: hdfs://path-redacted --> > s3a://path-redacted > at > org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:284) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:252) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: > java.io.IOException: Couldn't run retriable-command: Copying > hdfs://path-redacted to s3a://path-redacted > at > org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) > > at > org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:280) > > ... 10 more > Caused by: com.amazonaws.AmazonClientException: Unable to complete transfer: > timeout value is negative > at > com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:300) > at > com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:284) > at > com.amazonaws.services.s3.transfer.internal.CopyImpl.waitForCopyResult(CopyImpl.java:67) > > at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:943) > at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:357) > at > org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220) > at > org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:137) > > at > org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:100) > at > org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) > > ... 11 more > Caused by: java.lang.IllegalArgumentException: timeout value is negative > at java.lang.Thread.sleep(Native Method) > at > com.amazonaws.http.AmazonHttpClient.pauseBeforeNextRetry(AmazonHttpClient.java:864) > at > com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:353) > at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) > at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) > at > com.amazonaws.services.s3.AmazonS3Client.copyObject(AmazonS3Client.java:1507) > at > com.amazonaws.services.s3.transfer.internal.CopyCallable.copyInOneChunk(CopyCallable.java:143) > at > com.amazonaws.services.s3.transfer.internal.CopyCallable.call(CopyCallable.java:131) > > at > com.amazonaws.services.s3.transfer.internal.CopyMonitor.copy(CopyMonitor.java:189) > > at > com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:134) > > at > com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:46) > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12267) s3a failure due to integer overflow bug in AWS SDK
[ https://issues.apache.org/jira/browse/HADOOP-12267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-12267: -- Attachment: (was: HADOOP-12267.2.6.0.001-patch) > s3a failure due to integer overflow bug in AWS SDK > -- > > Key: HADOOP-12267 > URL: https://issues.apache.org/jira/browse/HADOOP-12267 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.6.0 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri > Attachments: HADOOP-12267.2.6.0.001.patch, > HADOOP-12267.2.7.1.001.patch > > > Under high load writing to Amazon AWS S3 storage, a client can be throttled > enough to encounter 24 retries in a row. > The amazon http client code (in aws-java-sdk jar) has a bug in its > exponential backoff retry code, that causes integer overflow, and a call to > Thread.sleep() with a negative value, which causes client to bail out with an > exception (see below). > Bug has been fixed in aws-java-sdk: > https://github.com/aws/aws-sdk-java/pull/388 > We need to pick this up for hadoop-tools/hadoop-aws. > Error: java.io.IOException: File copy failed: hdfs://path-redacted --> > s3a://path-redacted > at > org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:284) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:252) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: > java.io.IOException: Couldn't run retriable-command: Copying > hdfs://path-redacted to s3a://path-redacted > at > org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) > > at > org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:280) > > ... 10 more > Caused by: com.amazonaws.AmazonClientException: Unable to complete transfer: > timeout value is negative > at > com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:300) > at > com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:284) > at > com.amazonaws.services.s3.transfer.internal.CopyImpl.waitForCopyResult(CopyImpl.java:67) > > at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:943) > at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:357) > at > org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220) > at > org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:137) > > at > org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:100) > at > org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) > > ... 11 more > Caused by: java.lang.IllegalArgumentException: timeout value is negative > at java.lang.Thread.sleep(Native Method) > at > com.amazonaws.http.AmazonHttpClient.pauseBeforeNextRetry(AmazonHttpClient.java:864) > at > com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:353) > at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) > at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) > at > com.amazonaws.services.s3.AmazonS3Client.copyObject(AmazonS3Client.java:1507) > at > com.amazonaws.services.s3.transfer.internal.CopyCallable.copyInOneChunk(CopyCallable.java:143) > at > com.amazonaws.services.s3.transfer.internal.CopyCallable.call(CopyCallable.java:131) > > at > com.amazonaws.services.s3.transfer.internal.CopyMonitor.copy(CopyMonitor.java:189) > > at > com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:134) > > at > com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:46) > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12267) s3a failure due to integer overflow bug in AWS SDK
[ https://issues.apache.org/jira/browse/HADOOP-12267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639980#comment-14639980 ] Aaron Fabbri commented on HADOOP-12267: --- Bug fix was backported to AWS SDK 1.7.14. Officially, only last two releases are supported by Amazon. Currently this is 1.10.x and 1.9.x. I suggest 1.7.14 SDK jar for 2.6.x and 2.7.x, and then moving to latest/greatest 1.10.x for trunk. Adding patches. I tested the patches with some basic hdfs fs s3a:// commands. > s3a failure due to integer overflow bug in AWS SDK > -- > > Key: HADOOP-12267 > URL: https://issues.apache.org/jira/browse/HADOOP-12267 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.6.0 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri > Attachments: HADOOP-12267.2.6.0.001.patch, > HADOOP-12267.2.7.1.001.patch > > > Under high load writing to Amazon AWS S3 storage, a client can be throttled > enough to encounter 24 retries in a row. > The amazon http client code (in aws-java-sdk jar) has a bug in its > exponential backoff retry code, that causes integer overflow, and a call to > Thread.sleep() with a negative value, which causes client to bail out with an > exception (see below). > Bug has been fixed in aws-java-sdk: > https://github.com/aws/aws-sdk-java/pull/388 > We need to pick this up for hadoop-tools/hadoop-aws. > Error: java.io.IOException: File copy failed: hdfs://path-redacted --> > s3a://path-redacted > at > org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:284) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:252) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: > java.io.IOException: Couldn't run retriable-command: Copying > hdfs://path-redacted to s3a://path-redacted > at > org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) > > at > org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:280) > > ... 10 more > Caused by: com.amazonaws.AmazonClientException: Unable to complete transfer: > timeout value is negative > at > com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:300) > at > com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:284) > at > com.amazonaws.services.s3.transfer.internal.CopyImpl.waitForCopyResult(CopyImpl.java:67) > > at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:943) > at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:357) > at > org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220) > at > org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:137) > > at > org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:100) > at > org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) > > ... 11 more > Caused by: java.lang.IllegalArgumentException: timeout value is negative > at java.lang.Thread.sleep(Native Method) > at > com.amazonaws.http.AmazonHttpClient.pauseBeforeNextRetry(AmazonHttpClient.java:864) > at > com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:353) > at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) > at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) > at > com.amazonaws.services.s3.AmazonS3Client.copyObject(AmazonS3Client.java:1507) > at > com.amazonaws.services.s3.transfer.internal.CopyCallable.copyInOneChunk(CopyCallable.java:143) > at > com.amazonaws.services.s3.transfer.internal.CopyCallable.call(CopyCallable.java:131) > > at > com.amazonaws.services.s3.transfer.internal.CopyMonitor.copy(CopyMonitor.java:189) > > at > com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:134) > > at > com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:46) > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
