[jira] [Commented] (HADOOP-14521) KMS client needs retry logic
[
https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047443#comment-16047443
]
Hadoop QA commented on HADOOP-14521:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
24s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 19
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
4s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 29s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}153m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestAclsEndToEnd |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14521 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872794/HDFS-11804-trunk-7.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux da44da7a9e61 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / bec79ca |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12525/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12525/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
|
[jira] [Commented] (HADOOP-14503) Make RollingAverages a mutable metric
[ https://issues.apache.org/jira/browse/HADOOP-14503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047393#comment-16047393 ] Hudson commented on HADOOP-14503: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11862 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11862/]) HADOOP-14503. Make RollingAverages a mutable metric. Contributed by (arp: rev 8633ef8e10a78883fbd6bf197007dc5191bf4535) * (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableRollingAverages.java * (delete) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/RollingAverages.java * (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/MetricsTestHelper.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableMetricsFactory.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/metrics/TestDataNodeOutlierDetectionViaMetrics.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestHdfsConfigFields.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodePeerMetrics.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * (delete) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestRollingAverages.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodePeerMetrics.java * (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMutableRollingAverages.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MetricsRegistry.java > Make RollingAverages a mutable metric > - > > Key: HADOOP-14503 > URL: https://issues.apache.org/jira/browse/HADOOP-14503 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru > Fix For: 3.0.0-alpha4 > > Attachments: HADOOP-14503.001.patch, HADOOP-14503.002.patch, > HADOOP-14503.003.patch, HADOOP-14503.004.patch, HADOOP-14503.005.patch, > HADOOP-14503.006.patch, HADOOP-14503.007.patch > > > RollingAverages metric extends on MutableRatesWithAggregation metric and > maintains a group of rolling average metrics. This class should be allowed to > register as a metric with the MetricSystem. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, causing recursively list action endless
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongyuan Li updated HADOOP-14469:
-
Summary: FTPFileSystem#listStatus get currentPath and parentPath at the
same time, causing recursively list action endless (was:
FTPFileSystem#listStatus get currentPath and parentPath at the same time,
causing recursively list action stuck)
> FTPFileSystem#listStatus get currentPath and parentPath at the same time,
> causing recursively list action endless
> -
>
> Key: HADOOP-14469
> URL: https://issues.apache.org/jira/browse/HADOOP-14469
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, tools/distcp
>Affects Versions: 3.0.0-alpha3
> Environment: ftp build by windows7 + Serv-U_64 12.1.0.8
> code runs any os
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Critical
> Attachments: HADOOP-14469-001.patch, HADOOP-14469-002.patch,
> HADOOP-14469-003.patch
>
>
> for some ftpsystems, liststatus method will return new Path(".") and new
> Path(".."), thus causing list op looping.for example, Serv-U
> We can see the logic in code below:
> {code}
> private FileStatus[] listStatus(FTPClient client, Path file)
> throws IOException {
> ……
> FileStatus[] fileStats = new FileStatus[ftpFiles.length];
> for (int i = 0; i < ftpFiles.length; i++) {
> fileStats[i] = getFileStatus(ftpFiles[i], absolute);
> }
> return fileStats;
> }
> {code}
> {code}
> public void test() throws Exception{
> FTPFileSystem ftpFileSystem = new FTPFileSystem();
> ftpFileSystem.initialize(new
> Path("ftp://test:123456@192.168.44.1/;).toUri(),
> new Configuration());
> FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
> for(FileStatus fileStatus1 : fileStatus)
> System.out.println(fileStatus1);
> }
> {code}
> using test code below, the test results list below
> {code}
> FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
> isDirectory=false; length=2036; replication=1; blocksize=4096;
> modification_time=149579778; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
> isDirectory=false; length=1322; replication=1; blocksize=4096;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
> isDirectory=true; modification_time=149579712; access_time=0; owner=user;
> group=group; permission=-; isSymlink=false}
> {code}
> In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}}
> is obviously the current Path, and
> {{FileStatus{path=ftp://test:123456@192.168.44.1/;……}} is obviously the
> parent Path.
> So, if we want to walk the directory recursively, it will stuck.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, causing recursively list action stuck
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongyuan Li updated HADOOP-14469:
-
Description:
for some ftpsystems, liststatus method will return new Path(".") and new
Path(".."), thus causing list op looping.for example, Serv-U
We can see the logic in code below:
{code}
private FileStatus[] listStatus(FTPClient client, Path file)
throws IOException {
……
FileStatus[] fileStats = new FileStatus[ftpFiles.length];
for (int i = 0; i < ftpFiles.length; i++) {
fileStats[i] = getFileStatus(ftpFiles[i], absolute);
}
return fileStats;
}
{code}
{code}
public void test() throws Exception{
FTPFileSystem ftpFileSystem = new FTPFileSystem();
ftpFileSystem.initialize(new
Path("ftp://test:123456@192.168.44.1/;).toUri(),
new Configuration());
FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
for(FileStatus fileStatus1 : fileStatus)
System.out.println(fileStatus1);
}
{code}
using test code below, the test results list below
{code}
FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
modification_time=149671698; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
modification_time=149671698; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
modification_time=149671698; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
isDirectory=false; length=2036; replication=1; blocksize=4096;
modification_time=149579778; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
isDirectory=false; length=1322; replication=1; blocksize=4096;
modification_time=149671698; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
isDirectory=true; modification_time=149579712; access_time=0; owner=user;
group=group; permission=-; isSymlink=false}
{code}
In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}} is
obviously the current Path, and
{{FileStatus{path=ftp://test:123456@192.168.44.1/;……}} is obviously the parent
Path.
So, if we want to walk the directory recursively, it will stuck.
was:
for some ftpsystems, liststatus method will return new Path(".") and new
Path(".."), thus causing list op looping.for example, Serv-U
We can see the logic in code below:
{code}
private FileStatus[] listStatus(FTPClient client, Path file)
throws IOException {
……
FileStatus[] fileStats = new FileStatus[ftpFiles.length];
for (int i = 0; i < ftpFiles.length; i++) {
fileStats[i] = getFileStatus(ftpFiles[i], absolute);
}
return fileStats;
}
{code}
{code}
public void test() throws Exception{
FTPFileSystem ftpFileSystem = new FTPFileSystem();
ftpFileSystem.initialize(new
Path("ftp://test:123456@192.168.44.1/;).toUri(),
new Configuration());
FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
for(FileStatus fileStatus1 : fileStatus)
System.out.println(fileStatus1);
}
{code}
using test code below, the test results list below
{code}
FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
modification_time=149671698; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
modification_time=149671698; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
modification_time=149671698; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
isDirectory=false; length=2036; replication=1; blocksize=4096;
modification_time=149579778; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
isDirectory=false; length=1322; replication=1; blocksize=4096;
modification_time=149671698; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
isDirectory=true; modification_time=149579712; access_time=0; owner=user;
group=group; permission=-; isSymlink=false}
{code}
In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}} is
obviously the current Path, and
[jira] [Updated] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, causing recursively list action stuck
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongyuan Li updated HADOOP-14469:
-
Description:
for some ftpsystems, liststatus method will return new Path(".") and new
Path(".."), thus causing list op looping.for example, Serv-U
We can see the logic in code below:
{code}
private FileStatus[] listStatus(FTPClient client, Path file)
throws IOException {
……
FileStatus[] fileStats = new FileStatus[ftpFiles.length];
for (int i = 0; i < ftpFiles.length; i++) {
fileStats[i] = getFileStatus(ftpFiles[i], absolute);
}
return fileStats;
}
{code}
{code}
public void test() throws Exception{
FTPFileSystem ftpFileSystem = new FTPFileSystem();
ftpFileSystem.initialize(new
Path("ftp://test:123456@192.168.44.1/;).toUri(),
new Configuration());
FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
for(FileStatus fileStatus1 : fileStatus)
System.out.println(fileStatus1);
}
{code}
using test code below, the test results list below
{code}
FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
modification_time=149671698; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
modification_time=149671698; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
modification_time=149671698; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
isDirectory=false; length=2036; replication=1; blocksize=4096;
modification_time=149579778; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
isDirectory=false; length=1322; replication=1; blocksize=4096;
modification_time=149671698; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
isDirectory=true; modification_time=149579712; access_time=0; owner=user;
group=group; permission=-; isSymlink=false}
{code}
In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}} is
obviously the current Path, and
{{FileStatus{path=ftp://test:123456@192.168.44.1/;…… }} is obviously the
parent Path.
So, if we want to walk the directory recursively, it will stuck.
was:
for some ftpsystems, liststatus method will return new Path(".") and new
Path(".."), thus causing list op looping.for example, Serv-U
We can see the logic in code below:
{code}
private FileStatus[] listStatus(FTPClient client, Path file)
throws IOException {
……
FileStatus[] fileStats = new FileStatus[ftpFiles.length];
for (int i = 0; i < ftpFiles.length; i++) {
fileStats[i] = getFileStatus(ftpFiles[i], absolute);
}
return fileStats;
}
{code}
{code}
public void test() throws Exception{
FTPFileSystem ftpFileSystem = new FTPFileSystem();
ftpFileSystem.initialize(new
Path("ftp://test:123456@192.168.44.1/;).toUri(),
new Configuration());
FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
for(FileStatus fileStatus1 : fileStatus)
System.out.println(fileStatus1);
}
{code}
using test code below, the test results list below
{code}
FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
modification_time=149671698; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
modification_time=149671698; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
modification_time=149671698; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
isDirectory=false; length=2036; replication=1; blocksize=4096;
modification_time=149579778; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
isDirectory=false; length=1322; replication=1; blocksize=4096;
modification_time=149671698; access_time=0; owner=user; group=group;
permission=-; isSymlink=false}
FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
isDirectory=true; modification_time=149579712; access_time=0; owner=user;
group=group; permission=-; isSymlink=false}
{code}
In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}} is
obviously the current Path, and
[jira] [Updated] (HADOOP-14503) Make RollingAverages a mutable metric
[ https://issues.apache.org/jira/browse/HADOOP-14503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-14503: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha4 Status: Resolved (was: Patch Available) I've committed this and fixed the checkstyle whitespace issue. Thank you for the contribution [~hanishakoneru]. I hit some merge conflicts while applying the patch to branch-2. If you want to post a branch-2 patch I'd be happy to commit that too. > Make RollingAverages a mutable metric > - > > Key: HADOOP-14503 > URL: https://issues.apache.org/jira/browse/HADOOP-14503 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru > Fix For: 3.0.0-alpha4 > > Attachments: HADOOP-14503.001.patch, HADOOP-14503.002.patch, > HADOOP-14503.003.patch, HADOOP-14503.004.patch, HADOOP-14503.005.patch, > HADOOP-14503.006.patch, HADOOP-14503.007.patch > > > RollingAverages metric extends on MutableRatesWithAggregation metric and > maintains a group of rolling average metrics. This class should be allowed to > register as a metric with the MetricSystem. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14521) KMS client needs retry logic
[
https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047382#comment-16047382
]
Xiao Chen commented on HADOOP-14521:
Thanks for sticking with this Rushabh.
bq. From this comment, point 1,
Right... sorry I missed that one.
But what confuses me is that, what if {{maxNumRetries < providers.length}} ? If
someone configures it to be 2, and we have 3 providers, the call will fail and
print a message:
{code}
LOG.warn("Aborting since the Request has failed with all KMS"
+ " providers in the group or the exception is not recoverable.");
{code}
which isn't entirely accurate, because here it did not fail with all providers.
We can either update the message, or update the maxNumRetries to
providers.length to make sure we try at least once on all providers. Either way
feels okay to me. Slightly prefer the latter since by assumption all providers
should work and we don't have retry delay for the whole sweep.
Besides that, I'm +1 on patch 7.
> KMS client needs retry logic
>
>
> Key: HADOOP-14521
> URL: https://issues.apache.org/jira/browse/HADOOP-14521
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Attachments: HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch,
> HDFS-11804-trunk-3.patch, HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch,
> HDFS-11804-trunk-6.patch, HDFS-11804-trunk-7.patch, HDFS-11804-trunk.patch
>
>
> The kms client appears to have no retry logic – at all. It's completely
> decoupled from the ipc retry logic. This has major impacts if the KMS is
> unreachable for any reason, including but not limited to network connection
> issues, timeouts, the +restart during an upgrade+.
> This has some major ramifications:
> # Jobs may fail to submit, although oozie resubmit logic should mask it
> # Non-oozie launchers may experience higher rates if they do not already have
> retry logic.
> # Tasks reading EZ files will fail, probably be masked by framework reattempts
> # EZ file creation fails after creating a 0-length file – client receives
> EDEK in the create response, then fails when decrypting the EDEK
> # Bulk hadoop fs copies, and maybe distcp, will prematurely fail
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, causing recursively list action stuck
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongyuan Li updated HADOOP-14469:
-
Summary: FTPFileSystem#listStatus get currentPath and parentPath at the
same time, causing recursively list action stuck (was:
FTPFileSystem#listStatus get currentPath and parentPath at the same time, which
will cause recursively list action stuck)
> FTPFileSystem#listStatus get currentPath and parentPath at the same time,
> causing recursively list action stuck
> ---
>
> Key: HADOOP-14469
> URL: https://issues.apache.org/jira/browse/HADOOP-14469
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, tools/distcp
>Affects Versions: 3.0.0-alpha3
> Environment: ftp build by windows7 + Serv-U_64 12.1.0.8
> code runs any os
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Critical
> Attachments: HADOOP-14469-001.patch, HADOOP-14469-002.patch,
> HADOOP-14469-003.patch
>
>
> for some ftpsystems, liststatus method will return new Path(".") and new
> Path(".."), thus causing list op looping.for example, Serv-U
> We can see the logic in code below:
> {code}
> private FileStatus[] listStatus(FTPClient client, Path file)
> throws IOException {
> ……
> FileStatus[] fileStats = new FileStatus[ftpFiles.length];
> for (int i = 0; i < ftpFiles.length; i++) {
> fileStats[i] = getFileStatus(ftpFiles[i], absolute);
> }
> return fileStats;
> }
> {code}
> {code}
> public void test() throws Exception{
> FTPFileSystem ftpFileSystem = new FTPFileSystem();
> ftpFileSystem.initialize(new
> Path("ftp://test:123456@192.168.44.1/;).toUri(),
> new Configuration());
> FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
> for(FileStatus fileStatus1 : fileStatus)
> System.out.println(fileStatus1);
> }
> {code}
> using test code below, the test results list below
> {code}
> FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
> isDirectory=false; length=2036; replication=1; blocksize=4096;
> modification_time=149579778; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
> isDirectory=false; length=1322; replication=1; blocksize=4096;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
> isDirectory=true; modification_time=149579712; access_time=0; owner=user;
> group=group; permission=-; isSymlink=false}
> {code}
> In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}}
> is obviously the current Path, and
> {{FileStatus{path=ftp://test:123456@192.168.44.1/;…… }} is obviously the
> parent Path.
> So, if we want to walk the directory recursively, it will stuck.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, which will cause recursively list action stuck
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongyuan Li updated HADOOP-14469:
-
Summary: FTPFileSystem#listStatus get currentPath and parentPath at the
same time, which will cause recursively list action stuck (was:
FTPFileSystem#listStatus get currentPath and parentPath at the same time, which
will cause recursively list stuck)
> FTPFileSystem#listStatus get currentPath and parentPath at the same time,
> which will cause recursively list action stuck
>
>
> Key: HADOOP-14469
> URL: https://issues.apache.org/jira/browse/HADOOP-14469
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, tools/distcp
>Affects Versions: 3.0.0-alpha3
> Environment: ftp build by windows7 + Serv-U_64 12.1.0.8
> code runs any os
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Critical
> Attachments: HADOOP-14469-001.patch, HADOOP-14469-002.patch,
> HADOOP-14469-003.patch
>
>
> for some ftpsystems, liststatus method will return new Path(".") and new
> Path(".."), thus causing list op looping.for example, Serv-U
> We can see the logic in code below:
> {code}
> private FileStatus[] listStatus(FTPClient client, Path file)
> throws IOException {
> ……
> FileStatus[] fileStats = new FileStatus[ftpFiles.length];
> for (int i = 0; i < ftpFiles.length; i++) {
> fileStats[i] = getFileStatus(ftpFiles[i], absolute);
> }
> return fileStats;
> }
> {code}
> {code}
> public void test() throws Exception{
> FTPFileSystem ftpFileSystem = new FTPFileSystem();
> ftpFileSystem.initialize(new
> Path("ftp://test:123456@192.168.44.1/;).toUri(),
> new Configuration());
> FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
> for(FileStatus fileStatus1 : fileStatus)
> System.out.println(fileStatus1);
> }
> {code}
> using test code below, the test results list below
> {code}
> FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
> isDirectory=false; length=2036; replication=1; blocksize=4096;
> modification_time=149579778; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
> isDirectory=false; length=1322; replication=1; blocksize=4096;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
> isDirectory=true; modification_time=149579712; access_time=0; owner=user;
> group=group; permission=-; isSymlink=false}
> {code}
> In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}}
> is obviously the current Path, and
> {{FileStatus{path=ftp://test:123456@192.168.44.1/;…… }} is obviously the
> parent Path.
> So, if we want to walk the directory recursively, it will stuck.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14299) Hadoop Renew Thread for proxy users
[ https://issues.apache.org/jira/browse/HADOOP-14299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047371#comment-16047371 ] Hongyuan Li commented on HADOOP-14299: -- [~Prabhu Joseph] any updates about this? > Hadoop Renew Thread for proxy users > --- > > Key: HADOOP-14299 > URL: https://issues.apache.org/jira/browse/HADOOP-14299 > Project: Hadoop Common > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.7.1 >Reporter: Prabhu Joseph > > Currently Hadoop Client has a separate renew thread which is created only for > Authentication type Kerberos and not for Proxy. So for proxy users, a yarn > client monitoring a long running job will fail after initial ticket lifetime > with GSS initiate failed unless there is a manual re-kinit. > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L1030 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14503) Make RollingAverages a mutable metric
[
https://issues.apache.org/jira/browse/HADOOP-14503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047368#comment-16047368
]
Hadoop QA commented on HADOOP-14503:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
28s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 19
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 3s{color} | {color:orange} root: The patch generated 3 new + 595 unchanged
- 3 fixed = 598 total (was 598) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
57s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 13s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}165m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14503 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872783/HADOOP-14503.007.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1cc0f3566a54 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b3d3ede |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12523/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12523/artifact/patchprocess/diff-checkstyle-root.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12523/artifact/patchprocess/whitespace-eol.txt
|
| unit |
[jira] [Comment Edited] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, which will cause recursively list stuck
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045372#comment-16045372
]
Hongyuan Li edited comment on HADOOP-14469 at 6/13/17 3:12 AM:
---
[~ste...@apache.org] added test code and environment.
ftp built by serv-U x64 12.1.0.8, whose os is win7 x64
was (Author: hongyuan li):
[~ste...@apache.org] added test code and environment.
> FTPFileSystem#listStatus get currentPath and parentPath at the same time,
> which will cause recursively list stuck
> -
>
> Key: HADOOP-14469
> URL: https://issues.apache.org/jira/browse/HADOOP-14469
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, tools/distcp
>Affects Versions: 3.0.0-alpha3
> Environment: ftp build by windows7 + Serv-U_64 12.1.0.8
> code runs any os
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Critical
> Attachments: HADOOP-14469-001.patch, HADOOP-14469-002.patch,
> HADOOP-14469-003.patch
>
>
> for some ftpsystems, liststatus method will return new Path(".") and new
> Path(".."), thus causing list op looping.for example, Serv-U
> We can see the logic in code below:
> {code}
> private FileStatus[] listStatus(FTPClient client, Path file)
> throws IOException {
> ……
> FileStatus[] fileStats = new FileStatus[ftpFiles.length];
> for (int i = 0; i < ftpFiles.length; i++) {
> fileStats[i] = getFileStatus(ftpFiles[i], absolute);
> }
> return fileStats;
> }
> {code}
> {code}
> public void test() throws Exception{
> FTPFileSystem ftpFileSystem = new FTPFileSystem();
> ftpFileSystem.initialize(new
> Path("ftp://test:123456@192.168.44.1/;).toUri(),
> new Configuration());
> FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
> for(FileStatus fileStatus1 : fileStatus)
> System.out.println(fileStatus1);
> }
> {code}
> using test code below, the test results list below
> {code}
> FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
> isDirectory=false; length=2036; replication=1; blocksize=4096;
> modification_time=149579778; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
> isDirectory=false; length=1322; replication=1; blocksize=4096;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
> isDirectory=true; modification_time=149579712; access_time=0; owner=user;
> group=group; permission=-; isSymlink=false}
> {code}
> In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}}
> is obviously the current Path, and
> {{FileStatus{path=ftp://test:123456@192.168.44.1/;…… }} is obviously the
> parent Path.
> So, if we want to walk the directory recursively, it will stuck.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14429) FTPFileSystem#getFsAction always returns FsAction.NONE
[
https://issues.apache.org/jira/browse/HADOOP-14429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047358#comment-16047358
]
Hadoop QA commented on HADOOP-14429:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
34s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 19
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} hadoop-common-project/hadoop-common generated 0 new
+ 18 unchanged - 1 fixed = 18 total (was 19) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 31s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.sftp.TestSFTPFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14429 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872789/HADOOP-14429-005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4bb0ab8eaae2 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / bec79ca |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12524/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12524/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12524/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12524/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FTPFileSystem#getFsAction always returns FsAction.NONE
> ---
>
> Key: HADOOP-14429
> URL: https://issues.apache.org/jira/browse/HADOOP-14429
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0-alpha2
>
[jira] [Comment Edited] (HADOOP-14521) KMS client needs retry logic
[ https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047355#comment-16047355 ] Rushabh S Shah edited comment on HADOOP-14521 at 6/13/17 2:52 AM: -- bq. For the retries, do you think maybe it's more intuitive to specify num of failovers for the whole provider set? >From [this comment, point >1|https://issues.apache.org/jira/browse/HADOOP-14521?focusedCommentId=16036988=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16036988], > I think [~daryn] had a good reason explaining why have fix number of retries >is a good idea. I kind of agreed with him. So keeping the fix number of >retries even in patch #7. bq. KMSCP: the first @Link seems redundant Fixed in v7 bq. Good to have some validations on the config params (e.g. numRetries > 0 etc.) Added few pre conditions checks for all the config params. bq.I'd revert the change that breaks instead of returning. Thanks a lot for catching that. Fixed that in v7 of the patch. was (Author: shahrs87): bq. For the retries, do you think maybe it's more intuitive to specify num of failovers for the whole provider set? >From [this comment, point >1|https://issues.apache.org/jira/browse/HADOOP-14521?focusedCommentId=16036988=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16036988], > I think [~daryn] had a good reason explaining why have fix number of retries >is a good idea. I kind of agreed with him. So keeping the fix number of >retries even in patch #7. bq. KMSCP: the first @Link seems redundant Fixed in v7 bq. Good to have some validations on the config params (e.g. numRetries > 0 etc.) Added few pre conditions checks for all the config params. > KMS client needs retry logic > > > Key: HADOOP-14521 > URL: https://issues.apache.org/jira/browse/HADOOP-14521 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch, > HDFS-11804-trunk-3.patch, HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch, > HDFS-11804-trunk-6.patch, HDFS-11804-trunk-7.patch, HDFS-11804-trunk.patch > > > The kms client appears to have no retry logic – at all. It's completely > decoupled from the ipc retry logic. This has major impacts if the KMS is > unreachable for any reason, including but not limited to network connection > issues, timeouts, the +restart during an upgrade+. > This has some major ramifications: > # Jobs may fail to submit, although oozie resubmit logic should mask it > # Non-oozie launchers may experience higher rates if they do not already have > retry logic. > # Tasks reading EZ files will fail, probably be masked by framework reattempts > # EZ file creation fails after creating a 0-length file – client receives > EDEK in the create response, then fails when decrypting the EDEK > # Bulk hadoop fs copies, and maybe distcp, will prematurely fail -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14521) KMS client needs retry logic
[ https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HADOOP-14521: Attachment: HDFS-11804-trunk-7.patch bq. For the retries, do you think maybe it's more intuitive to specify num of failovers for the whole provider set? >From [this comment, point >1|https://issues.apache.org/jira/browse/HADOOP-14521?focusedCommentId=16036988=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16036988], > I think [~daryn] had a good reason explaining why have fix number of retries >is a good idea. I kind of agreed with him. So keeping the fix number of >retries even in patch #7. bq. KMSCP: the first @Link seems redundant Fixed in v7 bq. Good to have some validations on the config params (e.g. numRetries > 0 etc.) Added few pre conditions checks for all the config params. > KMS client needs retry logic > > > Key: HADOOP-14521 > URL: https://issues.apache.org/jira/browse/HADOOP-14521 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch, > HDFS-11804-trunk-3.patch, HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch, > HDFS-11804-trunk-6.patch, HDFS-11804-trunk-7.patch, HDFS-11804-trunk.patch > > > The kms client appears to have no retry logic – at all. It's completely > decoupled from the ipc retry logic. This has major impacts if the KMS is > unreachable for any reason, including but not limited to network connection > issues, timeouts, the +restart during an upgrade+. > This has some major ramifications: > # Jobs may fail to submit, although oozie resubmit logic should mask it > # Non-oozie launchers may experience higher rates if they do not already have > retry logic. > # Tasks reading EZ files will fail, probably be masked by framework reattempts > # EZ file creation fails after creating a 0-length file – client receives > EDEK in the create response, then fails when decrypting the EDEK > # Bulk hadoop fs copies, and maybe distcp, will prematurely fail -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14521) KMS client needs retry logic
[ https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HADOOP-14521: Status: Patch Available (was: Open) > KMS client needs retry logic > > > Key: HADOOP-14521 > URL: https://issues.apache.org/jira/browse/HADOOP-14521 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch, > HDFS-11804-trunk-3.patch, HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch, > HDFS-11804-trunk-6.patch, HDFS-11804-trunk-7.patch, HDFS-11804-trunk.patch > > > The kms client appears to have no retry logic – at all. It's completely > decoupled from the ipc retry logic. This has major impacts if the KMS is > unreachable for any reason, including but not limited to network connection > issues, timeouts, the +restart during an upgrade+. > This has some major ramifications: > # Jobs may fail to submit, although oozie resubmit logic should mask it > # Non-oozie launchers may experience higher rates if they do not already have > retry logic. > # Tasks reading EZ files will fail, probably be masked by framework reattempts > # EZ file creation fails after creating a 0-length file – client receives > EDEK in the create response, then fails when decrypting the EDEK > # Bulk hadoop fs copies, and maybe distcp, will prematurely fail -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14521) KMS client needs retry logic
[ https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HADOOP-14521: Status: Open (was: Patch Available) Thanks [~daryn] [~xiaochen] for your valuable reviews. Canceling the patch to address your comments. > KMS client needs retry logic > > > Key: HADOOP-14521 > URL: https://issues.apache.org/jira/browse/HADOOP-14521 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch, > HDFS-11804-trunk-3.patch, HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch, > HDFS-11804-trunk-6.patch, HDFS-11804-trunk.patch > > > The kms client appears to have no retry logic – at all. It's completely > decoupled from the ipc retry logic. This has major impacts if the KMS is > unreachable for any reason, including but not limited to network connection > issues, timeouts, the +restart during an upgrade+. > This has some major ramifications: > # Jobs may fail to submit, although oozie resubmit logic should mask it > # Non-oozie launchers may experience higher rates if they do not already have > retry logic. > # Tasks reading EZ files will fail, probably be masked by framework reattempts > # EZ file creation fails after creating a 0-length file – client receives > EDEK in the create response, then fails when decrypting the EDEK > # Bulk hadoop fs copies, and maybe distcp, will prematurely fail -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14470) CommandWithDestination#create used redundant ternary operator
[
https://issues.apache.org/jira/browse/HADOOP-14470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongyuan Li updated HADOOP-14470:
-
Component/s: fs
> CommandWithDestination#create used redundant ternary operator
> ---
>
> Key: HADOOP-14470
> URL: https://issues.apache.org/jira/browse/HADOOP-14470
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, fs
>Affects Versions: 3.0.0-alpha3
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Trivial
> Attachments: HADOOP-14470-001.patch
>
>
> in if statement，the lazyPersist is always true, thus the ternary operator is
> redundant，
> {{lazyPersist == true}} in if statment, so {{lazyPersist ? 1 :
> getDefaultReplication(item.path)}} is redundant.
> related code like below, which is in
> {{org.apache.hadoop.fs.shell.CommandWithDestination}} lineNumber : 504 :
> {code:java}
>FSDataOutputStream create(PathData item, boolean lazyPersist,
> boolean direct)
> throws IOException {
> try {
> if (lazyPersist) { // in if stament, lazyPersist is always true
> ……
> return create(item.path,
> FsPermission.getFileDefault().applyUMask(
> FsPermission.getUMask(getConf())),
> createFlags,
> getConf().getInt(IO_FILE_BUFFER_SIZE_KEY,
> IO_FILE_BUFFER_SIZE_DEFAULT),
> lazyPersist ? 1 : getDefaultReplication(item.path),
> // *this is redundant*
> getDefaultBlockSize(),
> null,
> null);
> } else {
> return create(item.path, true);
> }
> } finally { // might have been created but stream was interrupted
> if (!direct) {
> deleteOnExit(item.path);
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14429) FTPFileSystem#getFsAction always returns FsAction.NONE
[
https://issues.apache.org/jira/browse/HADOOP-14429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047318#comment-16047318
]
Hongyuan Li commented on HADOOP-14429:
--
[~yzhangal] resubmit the patch 005 as you have said.Added empty line added
between different sections in the unit test.
> FTPFileSystem#getFsAction always returns FsAction.NONE
> ---
>
> Key: HADOOP-14429
> URL: https://issues.apache.org/jira/browse/HADOOP-14429
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0-alpha2
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
> Attachments: HADOOP-14429-001.patch, HADOOP-14429-002.patch,
> HADOOP-14429-003.patch, HADOOP-14429-004.patch, HADOOP-14429-005.patch
>
>
>
> {code}
> private FsAction getFsAction(int accessGroup, FTPFile ftpFile) {
> FsAction action = FsAction.NONE;
> if (ftpFile.hasPermission(accessGroup, FTPFile.READ_PERMISSION)) {
> action.or(FsAction.READ);
> }
> if (ftpFile.hasPermission(accessGroup, FTPFile.WRITE_PERMISSION)) {
> action.or(FsAction.WRITE);
> }
> if (ftpFile.hasPermission(accessGroup, FTPFile.EXECUTE_PERMISSION)) {
> action.or(FsAction.EXECUTE);
> }
> return action;
> }
> {code}
> from code above, we can see that the getFsAction method doesnot modify the
> action generated by FsAction action = FsAction.NONE,which means it return
> FsAction.NONE all the time;
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14429) FTPFileSystem#getFsAction always returns FsAction.NONE
[
https://issues.apache.org/jira/browse/HADOOP-14429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongyuan Li updated HADOOP-14429:
-
Attachment: HADOOP-14429-005.patch
> FTPFileSystem#getFsAction always returns FsAction.NONE
> ---
>
> Key: HADOOP-14429
> URL: https://issues.apache.org/jira/browse/HADOOP-14429
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0-alpha2
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
> Attachments: HADOOP-14429-001.patch, HADOOP-14429-002.patch,
> HADOOP-14429-003.patch, HADOOP-14429-004.patch, HADOOP-14429-005.patch
>
>
>
> {code}
> private FsAction getFsAction(int accessGroup, FTPFile ftpFile) {
> FsAction action = FsAction.NONE;
> if (ftpFile.hasPermission(accessGroup, FTPFile.READ_PERMISSION)) {
> action.or(FsAction.READ);
> }
> if (ftpFile.hasPermission(accessGroup, FTPFile.WRITE_PERMISSION)) {
> action.or(FsAction.WRITE);
> }
> if (ftpFile.hasPermission(accessGroup, FTPFile.EXECUTE_PERMISSION)) {
> action.or(FsAction.EXECUTE);
> }
> return action;
> }
> {code}
> from code above, we can see that the getFsAction method doesnot modify the
> action generated by FsAction action = FsAction.NONE,which means it return
> FsAction.NONE all the time;
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14486) TestSFTPFileSystem#testGetAccessTime test failure using openJDk 1.8.0
[
https://issues.apache.org/jira/browse/HADOOP-14486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongyuan Li updated HADOOP-14486:
-
Component/s: fs
> TestSFTPFileSystem#testGetAccessTime test failure using openJDk 1.8.0
> ---
>
> Key: HADOOP-14486
> URL: https://issues.apache.org/jira/browse/HADOOP-14486
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0-alpha4
> Environment: Ubuntu 14.04
> x86, ppc64le
> $ java -version
> openjdk version "1.8.0_111"
> OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-3~14.04.1-b14)
> OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
>Reporter: Sonia Garudi
>Assignee: Hongyuan Li
> Attachments: HADOOP-14486-001.patch
>
>
> The TestSFTPFileSystem#testGetAccessTime test fails consistently with the
> error below:
> {code}
> java.lang.AssertionError: expected:<1496496040072> but was:<149649604>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testGetAccessTime(TestSFTPFileSystem.java:319)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, which will cause recursively list stuck
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongyuan Li updated HADOOP-14469:
-
Component/s: tools/distcp
> FTPFileSystem#listStatus get currentPath and parentPath at the same time,
> which will cause recursively list stuck
> -
>
> Key: HADOOP-14469
> URL: https://issues.apache.org/jira/browse/HADOOP-14469
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, tools/distcp
>Affects Versions: 3.0.0-alpha3
> Environment: ftp build by windows7 + Serv-U_64 12.1.0.8
> code runs any os
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Critical
> Attachments: HADOOP-14469-001.patch, HADOOP-14469-002.patch,
> HADOOP-14469-003.patch
>
>
> for some ftpsystems, liststatus method will return new Path(".") and new
> Path(".."), thus causing list op looping.for example, Serv-U
> We can see the logic in code below:
> {code}
> private FileStatus[] listStatus(FTPClient client, Path file)
> throws IOException {
> ……
> FileStatus[] fileStats = new FileStatus[ftpFiles.length];
> for (int i = 0; i < ftpFiles.length; i++) {
> fileStats[i] = getFileStatus(ftpFiles[i], absolute);
> }
> return fileStats;
> }
> {code}
> {code}
> public void test() throws Exception{
> FTPFileSystem ftpFileSystem = new FTPFileSystem();
> ftpFileSystem.initialize(new
> Path("ftp://test:123456@192.168.44.1/;).toUri(),
> new Configuration());
> FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
> for(FileStatus fileStatus1 : fileStatus)
> System.out.println(fileStatus1);
> }
> {code}
> using test code below, the test results list below
> {code}
> FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
> isDirectory=false; length=2036; replication=1; blocksize=4096;
> modification_time=149579778; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
> isDirectory=false; length=1322; replication=1; blocksize=4096;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
> isDirectory=true; modification_time=149579712; access_time=0; owner=user;
> group=group; permission=-; isSymlink=false}
> {code}
> In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}}
> is obviously the current Path, and
> {{FileStatus{path=ftp://test:123456@192.168.44.1/;…… }} is obviously the
> parent Path.
> So, if we want to walk the directory recursively, it will stuck.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14486) TestSFTPFileSystem#testGetAccessTime test failure using openJDK 1.8.0
[
https://issues.apache.org/jira/browse/HADOOP-14486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongyuan Li updated HADOOP-14486:
-
Summary: TestSFTPFileSystem#testGetAccessTime test failure using openJDK
1.8.0(was: TestSFTPFileSystem#testGetAccessTime test failure using openJDk
1.8.0 )
> TestSFTPFileSystem#testGetAccessTime test failure using openJDK 1.8.0
> ---
>
> Key: HADOOP-14486
> URL: https://issues.apache.org/jira/browse/HADOOP-14486
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0-alpha4
> Environment: Ubuntu 14.04
> x86, ppc64le
> $ java -version
> openjdk version "1.8.0_111"
> OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-3~14.04.1-b14)
> OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
>Reporter: Sonia Garudi
>Assignee: Hongyuan Li
> Attachments: HADOOP-14486-001.patch
>
>
> The TestSFTPFileSystem#testGetAccessTime test fails consistently with the
> error below:
> {code}
> java.lang.AssertionError: expected:<1496496040072> but was:<149649604>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testGetAccessTime(TestSFTPFileSystem.java:319)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14503) Make RollingAverages a mutable metric
[ https://issues.apache.org/jira/browse/HADOOP-14503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047272#comment-16047272 ] Arpit Agarwal commented on HADOOP-14503: Thanks [~hanishakoneru]. +1 pending Jenkins. > Make RollingAverages a mutable metric > - > > Key: HADOOP-14503 > URL: https://issues.apache.org/jira/browse/HADOOP-14503 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru > Attachments: HADOOP-14503.001.patch, HADOOP-14503.002.patch, > HADOOP-14503.003.patch, HADOOP-14503.004.patch, HADOOP-14503.005.patch, > HADOOP-14503.006.patch, HADOOP-14503.007.patch > > > RollingAverages metric extends on MutableRatesWithAggregation metric and > maintains a group of rolling average metrics. This class should be allowed to > register as a metric with the MetricSystem. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14503) Make RollingAverages a mutable metric
[ https://issues.apache.org/jira/browse/HADOOP-14503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HADOOP-14503: Attachment: HADOOP-14503.007.patch Thanks [~arpitagarwal] for the review. I have addressed your comments in patch v07. The test failures are unrelated and pass locally. > Make RollingAverages a mutable metric > - > > Key: HADOOP-14503 > URL: https://issues.apache.org/jira/browse/HADOOP-14503 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru > Attachments: HADOOP-14503.001.patch, HADOOP-14503.002.patch, > HADOOP-14503.003.patch, HADOOP-14503.004.patch, HADOOP-14503.005.patch, > HADOOP-14503.006.patch, HADOOP-14503.007.patch > > > RollingAverages metric extends on MutableRatesWithAggregation metric and > maintains a group of rolling average metrics. This class should be allowed to > register as a metric with the MetricSystem. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14524) Make CryptoCodec Closeable so it can be cleaned up proactively
Xiao Chen created HADOOP-14524:
--
Summary: Make CryptoCodec Closeable so it can be cleaned up
proactively
Key: HADOOP-14524
URL: https://issues.apache.org/jira/browse/HADOOP-14524
Project: Hadoop Common
Issue Type: Improvement
Reporter: Xiao Chen
Assignee: Xiao Chen
See HADOOP-14523 for motivation. Credit to [~mi...@cloudera.com] for reporting
initially on HADOOP-14523.
Basically, the CryptoCodec class is not a closeable, but the
OpensslAesCtrCryptoCodec implementation of it contains a closeable member (the
Random object). Currently it is left for {{finalize()}} to clean up, this would
create problems if OpensslAesCtrCryptoCodec is used with OsSecureRandom, which
could let OS run out of FDs on {{/dev/urandom}} if too many codecs created.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14524) Make CryptoCodec Closeable so it can be cleaned up proactively
[
https://issues.apache.org/jira/browse/HADOOP-14524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14524:
---
Description:
See HADOOP-14523 for motivation. Credit to [~mi...@cloudera.com] for reporting
initially there.
Basically, the CryptoCodec class is not a closeable, but the
OpensslAesCtrCryptoCodec implementation of it contains a closeable member (the
Random object). Currently it is left for {{finalize()}} to clean up, this would
create problems if OpensslAesCtrCryptoCodec is used with OsSecureRandom, which
could let OS run out of FDs on {{/dev/urandom}} if too many codecs created.
was:
See HADOOP-14523 for motivation. Credit to [~mi...@cloudera.com] for reporting
initially on HADOOP-14523.
Basically, the CryptoCodec class is not a closeable, but the
OpensslAesCtrCryptoCodec implementation of it contains a closeable member (the
Random object). Currently it is left for {{finalize()}} to clean up, this would
create problems if OpensslAesCtrCryptoCodec is used with OsSecureRandom, which
could let OS run out of FDs on {{/dev/urandom}} if too many codecs created.
> Make CryptoCodec Closeable so it can be cleaned up proactively
> --
>
> Key: HADOOP-14524
> URL: https://issues.apache.org/jira/browse/HADOOP-14524
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>
> See HADOOP-14523 for motivation. Credit to [~mi...@cloudera.com] for
> reporting initially there.
> Basically, the CryptoCodec class is not a closeable, but the
> OpensslAesCtrCryptoCodec implementation of it contains a closeable member
> (the Random object). Currently it is left for {{finalize()}} to clean up,
> this would create problems if OpensslAesCtrCryptoCodec is used with
> OsSecureRandom, which could let OS run out of FDs on {{/dev/urandom}} if too
> many codecs created.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14523) OpensslAesCtrCryptoCodec.finalize() holds excessive amounts of memory
[
https://issues.apache.org/jira/browse/HADOOP-14523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen reassigned HADOOP-14523:
--
Assignee: Misha Dmitriev
> OpensslAesCtrCryptoCodec.finalize() holds excessive amounts of memory
> -
>
> Key: HADOOP-14523
> URL: https://issues.apache.org/jira/browse/HADOOP-14523
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>
> I recently analyzed JVM heap dumps from Hive running a big workload. Two
> excerpts from the analysis done with jxray (www.jxray.com) are given below.
> It turns out that nearly a half of live memory is taken by objects awaiting
> finalization, and the biggest offender among them is class
> OpensslAesCtrCryptoCodec:
> {code}
> 401,189K (39.7%) (1 of sun.misc.Cleaner)
> <-- Java Static: sun.misc.Cleaner.first
> 400,572K (39.6%) (14001 of
> org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec,
> org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager, java.util.jar.JarFile etc.)
> <-- j.l.r.Finalizer.referent <-- j.l.r.Finalizer.{next} <--
> sun.misc.Cleaner.next <-- sun.misc.Cleaner.{next} <-- Java Static:
> sun.misc.Cleaner.first
> 270,673K (26.8%) (2138 of org.apache.hadoop.mapred.JobConf)
> <-- org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec.conf <--
> j.l.r.Finalizer.referent <-- j.l.r.Finalizer.{next} <-- sun.misc.Cleaner.next
> <-- sun.misc.Cleaner.{next} <-- Java Static: sun.misc.Cleaner.first
> -
> 102,232K (10.1%) (1 of j.l.r.Finalizer)
> <-- Java Static: java.lang.ref.Finalizer.unfinalized
> 101,676K (10.1%) (8613 of
> org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec,
> java.util.zip.ZipFile$ZipFileInflaterInputStream,
> org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager etc.)
> <-- j.l.r.Finalizer.referent <-- j.l.r.Finalizer.{next} <-- Java Static:
> java.lang.ref.Finalizer.unfinalized
> {code}
> This heap dump was taken using 'jmap -dump:live', which forces the JVM to run
> full GC before dumping the heap. So we are already looking at the heap right
> after GC, and yet all these unfinalized objects are there. I think this
> happens because the JVM always runs only one finalization thread, and thus
> the queue of objects that need finalization may get processed too slowly. My
> understanding is that finalization works as follows:
> 1. When GC runs, it discovers that object x that overrides finalize() is
> unreachable.
> 2. x is added to the finalization queue. So technically x is still reachable,
> it occupies memory, and _all the objects that it references stay in memory as
> well_.
> 3. The finalization thread processes objects from the finalization queue
> serially, thus x may stay in memory for long time.
> 4. x.finalize() is invoked, then x is made unreachable. If x stayed in memory
> for long time, it's now in Old Gen of the heap, so only full GC can clean it
> up.
> 5. When full GC finally occurs, x gets cleaned up.
> So finalization is formally reliable, but in practice it's quite possible
> that a lot of unreachable, but unfinalized objects flood the memory. I guess
> we are seeing all these OpensslAesCtrCryptoCodec objects when they are in
> phase 3 above. And the really bad thing is that these objects in turn keep in
> memory a whole lot of other stuff, in particular JobConf objects. Such a
> JobConf has nothing to do with finalization, yet the GC cannot release it
> until the corresponding OpensslAesCtrCryptoCodec's is gone.
> Here is OpensslAesCtrCryptoCodec.finalize() method with my comments:
> {code}
> protected void finalize() throws Throwable {
> try {
> Closeable r = (Closeable) this.random;
> r.close(); // Relevant only when (random instanceof OsSecureRandom ==
> true)
> } catch (ClassCastException e) {
> }
> super.finalize(); // Not needed, no finalize() in superclasses
> }
> {code}
> So, finalize() in this class, that may keep in memory a whole tree of
> objects, is relevant only when this codec is configured to use OsSecureRandom
> class. The latter reads random bytes from the configured file, and needs
> finalization to close the input stream associated with that file.
> The suggested fix is to remove finalize() from OpensslAesCtrCryptoCodec and
> add it to the only class from this "family" that really needs it,
> OsSecureRandom. That will ensure that only OsSecureRandom objects (if/when
> they are used) stay in memory awaiting finalization, and no other, irrelevant
> objects.
> Note that this solution means that streams are still closed lazily. This, in
> principle, may cause its own problems. So the most reliable fix would be to
> call OsSecureRandom.close() explicitly when it's not needed anymore. But the
> above fix is a necessary
[jira] [Created] (HADOOP-14523) OpensslAesCtrCryptoCodec.finalize() holds excessive amounts of memory
Misha Dmitriev created HADOOP-14523:
---
Summary: OpensslAesCtrCryptoCodec.finalize() holds excessive
amounts of memory
Key: HADOOP-14523
URL: https://issues.apache.org/jira/browse/HADOOP-14523
Project: Hadoop Common
Issue Type: Improvement
Reporter: Misha Dmitriev
I recently analyzed JVM heap dumps from Hive running a big workload. Two
excerpts from the analysis done with jxray (www.jxray.com) are given below. It
turns out that nearly a half of live memory is taken by objects awaiting
finalization, and the biggest offender among them is class
OpensslAesCtrCryptoCodec:
{code}
401,189K (39.7%) (1 of sun.misc.Cleaner)
<-- Java Static: sun.misc.Cleaner.first
400,572K (39.6%) (14001 of org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec,
org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager, java.util.jar.JarFile etc.)
<-- j.l.r.Finalizer.referent <-- j.l.r.Finalizer.{next} <--
sun.misc.Cleaner.next <-- sun.misc.Cleaner.{next} <-- Java Static:
sun.misc.Cleaner.first
270,673K (26.8%) (2138 of org.apache.hadoop.mapred.JobConf)
<-- org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec.conf <--
j.l.r.Finalizer.referent <-- j.l.r.Finalizer.{next} <-- sun.misc.Cleaner.next
<-- sun.misc.Cleaner.{next} <-- Java Static: sun.misc.Cleaner.first
-
102,232K (10.1%) (1 of j.l.r.Finalizer)
<-- Java Static: java.lang.ref.Finalizer.unfinalized
101,676K (10.1%) (8613 of org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec,
java.util.zip.ZipFile$ZipFileInflaterInputStream,
org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager etc.)
<-- j.l.r.Finalizer.referent <-- j.l.r.Finalizer.{next} <-- Java Static:
java.lang.ref.Finalizer.unfinalized
{code}
This heap dump was taken using 'jmap -dump:live', which forces the JVM to run
full GC before dumping the heap. So we are already looking at the heap right
after GC, and yet all these unfinalized objects are there. I think this happens
because the JVM always runs only one finalization thread, and thus the queue of
objects that need finalization may get processed too slowly. My understanding
is that finalization works as follows:
1. When GC runs, it discovers that object x that overrides finalize() is
unreachable.
2. x is added to the finalization queue. So technically x is still reachable,
it occupies memory, and _all the objects that it references stay in memory as
well_.
3. The finalization thread processes objects from the finalization queue
serially, thus x may stay in memory for long time.
4. x.finalize() is invoked, then x is made unreachable. If x stayed in memory
for long time, it's now in Old Gen of the heap, so only full GC can clean it up.
5. When full GC finally occurs, x gets cleaned up.
So finalization is formally reliable, but in practice it's quite possible that
a lot of unreachable, but unfinalized objects flood the memory. I guess we are
seeing all these OpensslAesCtrCryptoCodec objects when they are in phase 3
above. And the really bad thing is that these objects in turn keep in memory a
whole lot of other stuff, in particular JobConf objects. Such a JobConf has
nothing to do with finalization, yet the GC cannot release it until the
corresponding OpensslAesCtrCryptoCodec's is gone.
Here is OpensslAesCtrCryptoCodec.finalize() method with my comments:
{code}
protected void finalize() throws Throwable {
try {
Closeable r = (Closeable) this.random;
r.close(); // Relevant only when (random instanceof OsSecureRandom == true)
} catch (ClassCastException e) {
}
super.finalize(); // Not needed, no finalize() in superclasses
}
{code}
So, finalize() in this class, that may keep in memory a whole tree of objects,
is relevant only when this codec is configured to use OsSecureRandom class. The
latter reads random bytes from the configured file, and needs finalization to
close the input stream associated with that file.
The suggested fix is to remove finalize() from OpensslAesCtrCryptoCodec and add
it to the only class from this "family" that really needs it, OsSecureRandom.
That will ensure that only OsSecureRandom objects (if/when they are used) stay
in memory awaiting finalization, and no other, irrelevant objects.
Note that this solution means that streams are still closed lazily. This, in
principle, may cause its own problems. So the most reliable fix would be to
call OsSecureRandom.close() explicitly when it's not needed anymore. But the
above fix is a necessary first step anyway, it will remove the most acute
problem with memory and will not make any other things worse than they
currently are.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Commented] (HADOOP-14521) KMS client needs retry logic
[
https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047207#comment-16047207
]
Xiao Chen commented on HADOOP-14521:
Thanks for revving Rushabh. Looks pretty close to me.
Thanks for the explanations about {{AuthenticatedException}}, makes sense. I
don't know about why the IOExceptions are wrapped either (except for guessing
they're tricks with the method signature). Agree it's not related to this jira.
Patch 6 comments below:
For the retries, do you think maybe it's more intuitive to specify num of
failovers for the whole provider set? (i.e. {{maxFailovers =
conf.getInt(KMS_CLIENT_FAILOVER_MAX_RETRIES_KEY) * providers.length}}. (Then we
can have the core-default.xml set to 1) Will leave the decision to you and
Daryn.
Nits:
- KMSCP: the first {{@Link}} seems redundant in {{This will always create a
@Link ...}}
- Good to have some validations on the config params (e.g. numRetries > 0 etc.)
> KMS client needs retry logic
>
>
> Key: HADOOP-14521
> URL: https://issues.apache.org/jira/browse/HADOOP-14521
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Attachments: HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch,
> HDFS-11804-trunk-3.patch, HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch,
> HDFS-11804-trunk-6.patch, HDFS-11804-trunk.patch
>
>
> The kms client appears to have no retry logic – at all. It's completely
> decoupled from the ipc retry logic. This has major impacts if the KMS is
> unreachable for any reason, including but not limited to network connection
> issues, timeouts, the +restart during an upgrade+.
> This has some major ramifications:
> # Jobs may fail to submit, although oozie resubmit logic should mask it
> # Non-oozie launchers may experience higher rates if they do not already have
> retry logic.
> # Tasks reading EZ files will fail, probably be masked by framework reattempts
> # EZ file creation fails after creating a 0-length file – client receives
> EDEK in the create response, then fails when decrypting the EDEK
> # Bulk hadoop fs copies, and maybe distcp, will prematurely fail
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14521) KMS client needs retry logic
[
https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047200#comment-16047200
]
Daryn Sharp commented on HADOOP-14521:
--
I'd revert the change that breaks instead of returning. Otherwise this final
condition: {{(ret == null && ex != null)}} can be faulty. If the method was
successfully invoked and it returned null, and it was a retry, the condition
will erroneously throw the pre-retry exception.
> KMS client needs retry logic
>
>
> Key: HADOOP-14521
> URL: https://issues.apache.org/jira/browse/HADOOP-14521
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Attachments: HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch,
> HDFS-11804-trunk-3.patch, HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch,
> HDFS-11804-trunk-6.patch, HDFS-11804-trunk.patch
>
>
> The kms client appears to have no retry logic – at all. It's completely
> decoupled from the ipc retry logic. This has major impacts if the KMS is
> unreachable for any reason, including but not limited to network connection
> issues, timeouts, the +restart during an upgrade+.
> This has some major ramifications:
> # Jobs may fail to submit, although oozie resubmit logic should mask it
> # Non-oozie launchers may experience higher rates if they do not already have
> retry logic.
> # Tasks reading EZ files will fail, probably be masked by framework reattempts
> # EZ file creation fails after creating a 0-length file – client receives
> EDEK in the create response, then fails when decrypting the EDEK
> # Bulk hadoop fs copies, and maybe distcp, will prematurely fail
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14457) create() does not notify metadataStore of parent directories or ensure they're not existing files
[
https://issues.apache.org/jira/browse/HADOOP-14457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047186#comment-16047186
]
Aaron Fabbri commented on HADOOP-14457:
---
Thanks for interesting discussion [~mackrorysd].
{quote}
A capabilities-based approach seems to me to be far more complicated
{quote}
I think capabilities APIs are good practice for any interfaces that have
optional features. They make things explicit and improve testability. IMO
FileSystem should have something like this as well (i.e. HADOOP-9565,
HDFS-11644 come to mind).
{quote}
than just deciding whether it is or isn't a MetadataStore implementations job
to track parent directories.
{quote}
By "parent directories", I assume you mean ancestor directories all the way to
the root. To be clear we have decided what MetadataStore's contract is.
create() is the problem here.
We could change this, adding requirement that for any {{put(path)}}, the
{{MetadataStore}} *must* account for all ancestor directories in the whole
path. Currently the contract is not a *must* but a *may*. Reasons for this:
1. runtime complexity. Consider an FS client doing this:
MetadataStore.put(/a/b/c/d)
.. put(/a/b/c)
.. put(/a/b)
.. put(/a)
if each put() is does work of writing ancestors, it is wasted (~2x round trips
for any DB client, or depth^2 if it is dumb). I want to add more MetadataStore
backends in the future, and I'm not sure they will have the same internal
requirement that our Dynamo schema does (i.e. that all ancestors *must* be
tracked anyways).
2. The FS client should be doing the ancestor enumeration anyways (see
mkdirs()). Why force all MetadataStore to repeat it?
MetadataStore is a trailing log of metadata modifications that have been made
to a FileSystem. We expect a FS to be able to tell us what is is doing, or it
is broken (thus the bug here). I don't think putting a band-aid on create() is
a compelling reason to add more requirements to any future MetadataStore
implementations. Create is still broken here (i.e. if file is an ancestor)
either way.
TL;DR we could add the requirement if it makes sense, but I don't think it buys
us much here. I also feel like the implicit ancestor creation semantics of
Hadoop FS was not the best decision and am not eager to have it spread to
MetadataStore interface.
> create() does not notify metadataStore of parent directories or ensure
> they're not existing files
> -
>
> Key: HADOOP-14457
> URL: https://issues.apache.org/jira/browse/HADOOP-14457
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14457-HADOOP-13345.001.patch,
> HADOOP-14457-HADOOP-13345.002.patch, HADOOP-14457-HADOOP-13345.003.patch,
> HADOOP-14457-HADOOP-13345.004.patch, HADOOP-14457-HADOOP-13345.005.patch,
> HADOOP-14457-HADOOP-13345.006.patch, HADOOP-14457-HADOOP-13345.007.patch,
> HADOOP-14457-HADOOP-13345.008.patch, HADOOP-14457-HADOOP-13345.009.patch
>
>
> Not a great test yet, but it at least reliably demonstrates the issue.
> LocalMetadataStore will sometimes erroneously report that a directory is
> empty with isAuthoritative = true when it *definitely* has children the
> metadatastore should know about. It doesn't appear to happen if the children
> are just directory. The fact that it's returning an empty listing is
> concerning, but the fact that it says it's authoritative *might* be a second
> bug.
> {code}
> diff --git
> a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
>
> b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
> index 78b3970..1821d19 100644
> ---
> a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
> +++
> b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
> @@ -965,7 +965,7 @@ public boolean hasMetadataStore() {
>}
>
>@VisibleForTesting
> - MetadataStore getMetadataStore() {
> + public MetadataStore getMetadataStore() {
> return metadataStore;
>}
>
> diff --git
> a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java
>
> b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java
> index 4339649..881bdc9 100644
> ---
> a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java
> +++
> b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java
> @@ -23,6 +23,11 @@
> import org.apache.hadoop.fs.contract.AbstractFSContract;
> import org.apache.hadoop.fs.FileSystem;
> import
[jira] [Commented] (HADOOP-14503) Make RollingAverages a mutable metric
[ https://issues.apache.org/jira/browse/HADOOP-14503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047161#comment-16047161 ] Arpit Agarwal commented on HADOOP-14503: Thanks [~hanishakoneru]. The patch lgtm. Can you please address the checkstyle issues? For the first checkstyle issue, you can add a private constructor to the MetricsTestHelper class and make it a final class. Also there is a pre-existing typo in DataNodePeerMetrics (sendPacketDownstreamRollingAvgerages). Could you please fix that too? > Make RollingAverages a mutable metric > - > > Key: HADOOP-14503 > URL: https://issues.apache.org/jira/browse/HADOOP-14503 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru > Attachments: HADOOP-14503.001.patch, HADOOP-14503.002.patch, > HADOOP-14503.003.patch, HADOOP-14503.004.patch, HADOOP-14503.005.patch, > HADOOP-14503.006.patch > > > RollingAverages metric extends on MutableRatesWithAggregation metric and > maintains a group of rolling average metrics. This class should be allowed to > register as a metric with the MetricSystem. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14501) Switch from aalto-xml to woodstox to handle odd XML features
[
https://issues.apache.org/jira/browse/HADOOP-14501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047151#comment-16047151
]
Hudson commented on HADOOP-14501:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11858 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11858/])
HADOOP-14501. Switch from aalto-xml to woodstox to handle odd XML (jeagles: rev
a81916ea89d59c1642b3462e3d7c6c1acb1e7109)
* (edit)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
* (edit) hadoop-project/pom.xml
* (edit) hadoop-common-project/hadoop-common/pom.xml
> Switch from aalto-xml to woodstox to handle odd XML features
>
>
> Key: HADOOP-14501
> URL: https://issues.apache.org/jira/browse/HADOOP-14501
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Affects Versions: 2.9.0, 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Jonathan Eagles
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-14501.1.patch, HADOOP-14501.2.patch,
> HADOOP-14501.3-branch-2.patch, HADOOP-14501.3.patch,
> HADOOP-14501.4-branch-2.patch, HADOOP-14501.4.patch, HADOOP-14501.5.patch,
> HADOOP-14501-branch-2.1.patch
>
>
> [~hgadre] tried testing solr with a Hadoop 3 client. He saw various test case
> failures due to what look like functionality gaps in the new aalto-xml stax
> implementation pulled in by HADOOP-14216:
> {noformat}
>[junit4]> Throwable #1: com.fasterxml.aalto.WFCException: Illegal XML
> character ('ü' (code 252))
>
>[junit4]> Caused by: com.fasterxml.aalto.WFCException: General entity
> reference () encountered in entity expanding mode: operation not (yet)
> implemented
> ...
>[junit4]> Throwable #1: org.apache.solr.common.SolrException: General
> entity reference () encountered in entity expanding mode: operation
> not (yet) implemented
> {noformat}
> These were from the following test case executions:
> {noformat}
> NOTE: reproduce with: ant test -Dtestcase=DocumentAnalysisRequestHandlerTest
> -Dtests.method=testCharsetOutsideDocument -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=und -Dtests.timezone=Atlantic/Faeroe
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> NOTE: reproduce with: ant test -Dtestcase=MBeansHandlerTest
> -Dtests.method=testXMLDiffWithExternalEntity -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=en-US -Dtests.timezone=US/Aleutian
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> NOTE: reproduce with: ant test -Dtestcase=XmlUpdateRequestHandlerTest
> -Dtests.method=testExternalEntities -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=hr -Dtests.timezone=America/Barbados
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> NOTE: reproduce with: ant test -Dtestcase=XmlUpdateRequestHandlerTest
> -Dtests.method=testNamedEntity -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=hr -Dtests.timezone=America/Barbados
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14501) Switch from aalto-xml to woodstox to handle odd XML features
[
https://issues.apache.org/jira/browse/HADOOP-14501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047141#comment-16047141
]
Jonathan Eagles commented on HADOOP-14501:
--
Thanks again, [~andrew.wang]!
> Switch from aalto-xml to woodstox to handle odd XML features
>
>
> Key: HADOOP-14501
> URL: https://issues.apache.org/jira/browse/HADOOP-14501
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Affects Versions: 2.9.0, 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Jonathan Eagles
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-14501.1.patch, HADOOP-14501.2.patch,
> HADOOP-14501.3-branch-2.patch, HADOOP-14501.3.patch,
> HADOOP-14501.4-branch-2.patch, HADOOP-14501.4.patch, HADOOP-14501.5.patch,
> HADOOP-14501-branch-2.1.patch
>
>
> [~hgadre] tried testing solr with a Hadoop 3 client. He saw various test case
> failures due to what look like functionality gaps in the new aalto-xml stax
> implementation pulled in by HADOOP-14216:
> {noformat}
>[junit4]> Throwable #1: com.fasterxml.aalto.WFCException: Illegal XML
> character ('ü' (code 252))
>
>[junit4]> Caused by: com.fasterxml.aalto.WFCException: General entity
> reference () encountered in entity expanding mode: operation not (yet)
> implemented
> ...
>[junit4]> Throwable #1: org.apache.solr.common.SolrException: General
> entity reference () encountered in entity expanding mode: operation
> not (yet) implemented
> {noformat}
> These were from the following test case executions:
> {noformat}
> NOTE: reproduce with: ant test -Dtestcase=DocumentAnalysisRequestHandlerTest
> -Dtests.method=testCharsetOutsideDocument -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=und -Dtests.timezone=Atlantic/Faeroe
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> NOTE: reproduce with: ant test -Dtestcase=MBeansHandlerTest
> -Dtests.method=testXMLDiffWithExternalEntity -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=en-US -Dtests.timezone=US/Aleutian
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> NOTE: reproduce with: ant test -Dtestcase=XmlUpdateRequestHandlerTest
> -Dtests.method=testExternalEntities -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=hr -Dtests.timezone=America/Barbados
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> NOTE: reproduce with: ant test -Dtestcase=XmlUpdateRequestHandlerTest
> -Dtests.method=testNamedEntity -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=hr -Dtests.timezone=America/Barbados
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14501) Switch from aalto-xml to woodstox to handle odd XML features
[
https://issues.apache.org/jira/browse/HADOOP-14501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated HADOOP-14501:
-
Resolution: Fixed
Fix Version/s: 3.0.0-alpha4
2.9.0
Status: Resolved (was: Patch Available)
> Switch from aalto-xml to woodstox to handle odd XML features
>
>
> Key: HADOOP-14501
> URL: https://issues.apache.org/jira/browse/HADOOP-14501
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Affects Versions: 2.9.0, 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Jonathan Eagles
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-14501.1.patch, HADOOP-14501.2.patch,
> HADOOP-14501.3-branch-2.patch, HADOOP-14501.3.patch,
> HADOOP-14501.4-branch-2.patch, HADOOP-14501.4.patch, HADOOP-14501.5.patch,
> HADOOP-14501-branch-2.1.patch
>
>
> [~hgadre] tried testing solr with a Hadoop 3 client. He saw various test case
> failures due to what look like functionality gaps in the new aalto-xml stax
> implementation pulled in by HADOOP-14216:
> {noformat}
>[junit4]> Throwable #1: com.fasterxml.aalto.WFCException: Illegal XML
> character ('ü' (code 252))
>
>[junit4]> Caused by: com.fasterxml.aalto.WFCException: General entity
> reference () encountered in entity expanding mode: operation not (yet)
> implemented
> ...
>[junit4]> Throwable #1: org.apache.solr.common.SolrException: General
> entity reference () encountered in entity expanding mode: operation
> not (yet) implemented
> {noformat}
> These were from the following test case executions:
> {noformat}
> NOTE: reproduce with: ant test -Dtestcase=DocumentAnalysisRequestHandlerTest
> -Dtests.method=testCharsetOutsideDocument -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=und -Dtests.timezone=Atlantic/Faeroe
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> NOTE: reproduce with: ant test -Dtestcase=MBeansHandlerTest
> -Dtests.method=testXMLDiffWithExternalEntity -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=en-US -Dtests.timezone=US/Aleutian
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> NOTE: reproduce with: ant test -Dtestcase=XmlUpdateRequestHandlerTest
> -Dtests.method=testExternalEntities -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=hr -Dtests.timezone=America/Barbados
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> NOTE: reproduce with: ant test -Dtestcase=XmlUpdateRequestHandlerTest
> -Dtests.method=testNamedEntity -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=hr -Dtests.timezone=America/Barbados
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14522) Unable to see logs for Job end notification
[
https://issues.apache.org/jira/browse/HADOOP-14522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Subhashrao Saley updated HADOOP-14522:
-
Labels: applicationmaster mapreduce (was: )
> Unable to see logs for Job end notification
> ---
>
> Key: HADOOP-14522
> URL: https://issues.apache.org/jira/browse/HADOOP-14522
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Satish Subhashrao Saley
> Labels: applicationmaster, mapreduce
>
> Hi, I have set {{mapreduce.job.end-notification.url}} and I am able to get
> the notification correctly. But I am unable to see log lines from class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster regarding job end
> notification. When I was trying out different notification urls (before
> stabilizing my app), I didn't have any way to look at the logs and wasted a
> lot of time figuring out actual issue. It would be good if these logs are
> also included in syslog for container.
> [relevant
> code|https://github.com/apache/hadoop/blob/branch-2.8/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java#L641-L650]
> {code}
> notifyIsLastAMRetry(isLastAMRetry);
> // Stop all services
> // This will also send the final report to the ResourceManager
> LOG.info("Calling stop for all the services"); < Able to see this.
> MRAppMaster.this.stop();
> if (isLastAMRetry) {
> // Send job-end notification when it is safe to report termination to
> // users and it is the last AM retry
> if (getConfig().get(MRJobConfig.MR_JOB_END_NOTIFICATION_URL) != null)
> {
> try {
> LOG.info("Job end notification started for jobID : "
> + job.getReport().getJobId()); <--- unable to see this
> JobEndNotifier notifier = new JobEndNotifier();
> notifier.setConf(getConfig());
> JobReport report = job.getReport();
> {code}
> Is it because we are shutting down all services including
> JobHistoryEventHandler in {{MRAppMaster.this.stop();}} ?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14522) Unable to see logs for Job end notification
[
https://issues.apache.org/jira/browse/HADOOP-14522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Subhashrao Saley updated HADOOP-14522:
-
Affects Version/s: 2.8.0
> Unable to see logs for Job end notification
> ---
>
> Key: HADOOP-14522
> URL: https://issues.apache.org/jira/browse/HADOOP-14522
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Satish Subhashrao Saley
>
> Hi, I have set {{mapreduce.job.end-notification.url}} and I am able to get
> the notification correctly. But I am unable to see log lines from class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster regarding job end
> notification. When I was trying out different notification urls (before
> stabilizing my app), I didn't have any way to look at the logs and wasted a
> lot of time figuring out actual issue. It would be good if these logs are
> also included in syslog for container.
> [relevant
> code|https://github.com/apache/hadoop/blob/branch-2.8/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java#L641-L650]
> {code}
> notifyIsLastAMRetry(isLastAMRetry);
> // Stop all services
> // This will also send the final report to the ResourceManager
> LOG.info("Calling stop for all the services"); < Able to see this.
> MRAppMaster.this.stop();
> if (isLastAMRetry) {
> // Send job-end notification when it is safe to report termination to
> // users and it is the last AM retry
> if (getConfig().get(MRJobConfig.MR_JOB_END_NOTIFICATION_URL) != null)
> {
> try {
> LOG.info("Job end notification started for jobID : "
> + job.getReport().getJobId()); <--- unable to see this
> JobEndNotifier notifier = new JobEndNotifier();
> notifier.setConf(getConfig());
> JobReport report = job.getReport();
> {code}
> Is it because we are shutting down all services including
> JobHistoryEventHandler in {{MRAppMaster.this.stop();}} ?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14522) Unable to see logs for Job end notification
Satish Subhashrao Saley created HADOOP-14522:
Summary: Unable to see logs for Job end notification
Key: HADOOP-14522
URL: https://issues.apache.org/jira/browse/HADOOP-14522
Project: Hadoop Common
Issue Type: Bug
Reporter: Satish Subhashrao Saley
Hi, I have set {{mapreduce.job.end-notification.url}} and I am able to get the
notification correctly. But I am unable to see log lines from class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster regarding job end notification.
When I was trying out different notification urls (before stabilizing my app),
I didn't have any way to look at the logs and wasted a lot of time figuring out
actual issue. It would be good if these logs are also included in syslog for
container.
[relevant
code|https://github.com/apache/hadoop/blob/branch-2.8/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java#L641-L650]
{code}
notifyIsLastAMRetry(isLastAMRetry);
// Stop all services
// This will also send the final report to the ResourceManager
LOG.info("Calling stop for all the services"); < Able to see this.
MRAppMaster.this.stop();
if (isLastAMRetry) {
// Send job-end notification when it is safe to report termination to
// users and it is the last AM retry
if (getConfig().get(MRJobConfig.MR_JOB_END_NOTIFICATION_URL) != null) {
try {
LOG.info("Job end notification started for jobID : "
+ job.getReport().getJobId()); <--- unable to see this
JobEndNotifier notifier = new JobEndNotifier();
notifier.setConf(getConfig());
JobReport report = job.getReport();
{code}
Is it because we are shutting down all services including
JobHistoryEventHandler in {{MRAppMaster.this.stop();}} ?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14520) Block compaction for WASB
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Georgi Chalakov updated HADOOP-14520: - Status: Patch Available (was: In Progress) > Block compaction for WASB > - > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Attachments: HADOOP-14520-01.patch, HADOOP-14520-01-test.txt > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above a predefined, configurable value, > next hflush/hsync triggers the block compaction process. Block compaction > replaces a sequence of blocks with one block. From all the sequences with > total length less than 4M, compaction chooses the longest one. It is a greedy > algorithm that preserve all potential candidates for the next round. Block > Compaction for WASB increases data durability and allows using block blobs > instead of page blobs. By default, block compaction is disabled. Similar to > the configuration for page blobs, the client needs to specify HDFS folders > where block compaction over block blobs is enabled. > Results for HADOOP-14520-01.patch > Tests run: 704, Failures: 0, Errors: 0, Skipped: 119 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14520) Block compaction for WASB
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Georgi Chalakov updated HADOOP-14520: - Status: In Progress (was: Patch Available) > Block compaction for WASB > - > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Attachments: HADOOP-14520-01.patch, HADOOP-14520-01-test.txt > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above a predefined, configurable value, > next hflush/hsync triggers the block compaction process. Block compaction > replaces a sequence of blocks with one block. From all the sequences with > total length less than 4M, compaction chooses the longest one. It is a greedy > algorithm that preserve all potential candidates for the next round. Block > Compaction for WASB increases data durability and allows using block blobs > instead of page blobs. By default, block compaction is disabled. Similar to > the configuration for page blobs, the client needs to specify HDFS folders > where block compaction over block blobs is enabled. > Results for HADOOP-14520-01.patch > Tests run: 704, Failures: 0, Errors: 0, Skipped: 119 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14520) Block compaction for WASB
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Georgi Chalakov updated HADOOP-14520: - Status: Patch Available (was: In Progress) > Block compaction for WASB > - > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Attachments: HADOOP-14520-01.patch, HADOOP-14520-01-test.txt > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above a predefined, configurable value, > next hflush/hsync triggers the block compaction process. Block compaction > replaces a sequence of blocks with one block. From all the sequences with > total length less than 4M, compaction chooses the longest one. It is a greedy > algorithm that preserve all potential candidates for the next round. Block > Compaction for WASB increases data durability and allows using block blobs > instead of page blobs. By default, block compaction is disabled. Similar to > the configuration for page blobs, the client needs to specify HDFS folders > where block compaction over block blobs is enabled. > Results for HADOOP-14520-01.patch > Tests run: 704, Failures: 0, Errors: 0, Skipped: 119 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14501) Switch from aalto-xml to woodstox to handle odd XML features
[
https://issues.apache.org/jira/browse/HADOOP-14501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047124#comment-16047124
]
Andrew Wang commented on HADOOP-14501:
--
Yea SGTM, thanks Jonathan!
> Switch from aalto-xml to woodstox to handle odd XML features
>
>
> Key: HADOOP-14501
> URL: https://issues.apache.org/jira/browse/HADOOP-14501
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Affects Versions: 2.9.0, 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Jonathan Eagles
>Priority: Blocker
> Attachments: HADOOP-14501.1.patch, HADOOP-14501.2.patch,
> HADOOP-14501.3-branch-2.patch, HADOOP-14501.3.patch,
> HADOOP-14501.4-branch-2.patch, HADOOP-14501.4.patch, HADOOP-14501.5.patch,
> HADOOP-14501-branch-2.1.patch
>
>
> [~hgadre] tried testing solr with a Hadoop 3 client. He saw various test case
> failures due to what look like functionality gaps in the new aalto-xml stax
> implementation pulled in by HADOOP-14216:
> {noformat}
>[junit4]> Throwable #1: com.fasterxml.aalto.WFCException: Illegal XML
> character ('ü' (code 252))
>
>[junit4]> Caused by: com.fasterxml.aalto.WFCException: General entity
> reference () encountered in entity expanding mode: operation not (yet)
> implemented
> ...
>[junit4]> Throwable #1: org.apache.solr.common.SolrException: General
> entity reference () encountered in entity expanding mode: operation
> not (yet) implemented
> {noformat}
> These were from the following test case executions:
> {noformat}
> NOTE: reproduce with: ant test -Dtestcase=DocumentAnalysisRequestHandlerTest
> -Dtests.method=testCharsetOutsideDocument -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=und -Dtests.timezone=Atlantic/Faeroe
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> NOTE: reproduce with: ant test -Dtestcase=MBeansHandlerTest
> -Dtests.method=testXMLDiffWithExternalEntity -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=en-US -Dtests.timezone=US/Aleutian
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> NOTE: reproduce with: ant test -Dtestcase=XmlUpdateRequestHandlerTest
> -Dtests.method=testExternalEntities -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=hr -Dtests.timezone=America/Barbados
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> NOTE: reproduce with: ant test -Dtestcase=XmlUpdateRequestHandlerTest
> -Dtests.method=testNamedEntity -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=hr -Dtests.timezone=America/Barbados
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14501) Switch from aalto-xml to woodstox to handle odd XML features
[
https://issues.apache.org/jira/browse/HADOOP-14501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047121#comment-16047121
]
Jonathan Eagles commented on HADOOP-14501:
--
[~andrew.wang], should I go ahead and check this into trunk and branch-2?
> Switch from aalto-xml to woodstox to handle odd XML features
>
>
> Key: HADOOP-14501
> URL: https://issues.apache.org/jira/browse/HADOOP-14501
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Affects Versions: 2.9.0, 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Jonathan Eagles
>Priority: Blocker
> Attachments: HADOOP-14501.1.patch, HADOOP-14501.2.patch,
> HADOOP-14501.3-branch-2.patch, HADOOP-14501.3.patch,
> HADOOP-14501.4-branch-2.patch, HADOOP-14501.4.patch, HADOOP-14501.5.patch,
> HADOOP-14501-branch-2.1.patch
>
>
> [~hgadre] tried testing solr with a Hadoop 3 client. He saw various test case
> failures due to what look like functionality gaps in the new aalto-xml stax
> implementation pulled in by HADOOP-14216:
> {noformat}
>[junit4]> Throwable #1: com.fasterxml.aalto.WFCException: Illegal XML
> character ('ü' (code 252))
>
>[junit4]> Caused by: com.fasterxml.aalto.WFCException: General entity
> reference () encountered in entity expanding mode: operation not (yet)
> implemented
> ...
>[junit4]> Throwable #1: org.apache.solr.common.SolrException: General
> entity reference () encountered in entity expanding mode: operation
> not (yet) implemented
> {noformat}
> These were from the following test case executions:
> {noformat}
> NOTE: reproduce with: ant test -Dtestcase=DocumentAnalysisRequestHandlerTest
> -Dtests.method=testCharsetOutsideDocument -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=und -Dtests.timezone=Atlantic/Faeroe
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> NOTE: reproduce with: ant test -Dtestcase=MBeansHandlerTest
> -Dtests.method=testXMLDiffWithExternalEntity -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=en-US -Dtests.timezone=US/Aleutian
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> NOTE: reproduce with: ant test -Dtestcase=XmlUpdateRequestHandlerTest
> -Dtests.method=testExternalEntities -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=hr -Dtests.timezone=America/Barbados
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> NOTE: reproduce with: ant test -Dtestcase=XmlUpdateRequestHandlerTest
> -Dtests.method=testNamedEntity -Dtests.seed=2F739D88D9C723CA
> -Dtests.slow=true -Dtests.locale=hr -Dtests.timezone=America/Barbados
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14503) Make RollingAverages a mutable metric
[
https://issues.apache.org/jira/browse/HADOOP-14503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047103#comment-16047103
]
Hadoop QA commented on HADOOP-14503:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
11s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
31s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 19
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
11s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 4s{color} | {color:orange} root: The patch generated 4 new + 595 unchanged
- 3 fixed = 599 total (was 598) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
32s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 59s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}161m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14503 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872763/HADOOP-14503.006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ced59ec04b28 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 86368cc |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12522/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12522/artifact/patchprocess/diff-checkstyle-root.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12522/artifact/patchprocess/whitespace-eol.txt
|
| unit
[jira] [Updated] (HADOOP-14520) Block compaction for WASB
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Georgi Chalakov updated HADOOP-14520: - Status: In Progress (was: Patch Available) > Block compaction for WASB > - > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Attachments: HADOOP-14520-01.patch, HADOOP-14520-01-test.txt > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above a predefined, configurable value, > next hflush/hsync triggers the block compaction process. Block compaction > replaces a sequence of blocks with one block. From all the sequences with > total length less than 4M, compaction chooses the longest one. It is a greedy > algorithm that preserve all potential candidates for the next round. Block > Compaction for WASB increases data durability and allows using block blobs > instead of page blobs. By default, block compaction is disabled. Similar to > the configuration for page blobs, the client needs to specify HDFS folders > where block compaction over block blobs is enabled. > Results for HADOOP-14520-01.patch > Tests run: 704, Failures: 0, Errors: 0, Skipped: 119 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14518) Customize User-Agent header sent in HTTP/HTTPS requests by WASB.
[ https://issues.apache.org/jira/browse/HADOOP-14518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Georgi Chalakov updated HADOOP-14518: - Target Version/s: 3.0.0-alpha3, 2.8.0 (was: 2.8.0, 3.0.0-alpha3) Status: In Progress (was: Patch Available) > Customize User-Agent header sent in HTTP/HTTPS requests by WASB. > > > Key: HADOOP-14518 > URL: https://issues.apache.org/jira/browse/HADOOP-14518 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov >Priority: Minor > Attachments: HADOOP-14518-01.patch, HADOOP-14518-01-test.txt, > HADOOP-14518-02.patch, HADOOP-14518-03.patch > > > WASB passes a User-Agent header to the Azure back-end. Right now, it uses the > default value set by the Azure Client SDK, so Hadoop traffic doesn't appear > any different from general Blob traffic. If we customize the User-Agent > header, then it will enable better troubleshooting and analysis by Azure > service. > The following configuration > > fs.azure.user.agent.id > MSFT > > set the user agent to > User-Agent: WASB/3.0.0-alpha4-SNAPSHOT (MSFT) Azure-Storage/4.2.0 > (JavaJRE 1.8.0_131; WindowsServer2012R2 6.3) > Test Results : > Tests run: 703, Failures: 0, Errors: 0, Skipped: 119 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14519) Client$Connection#waitForWork may suffer spurious wakeup
[
https://issues.apache.org/jira/browse/HADOOP-14519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047047#comment-16047047
]
Chen Liang edited comment on HADOOP-14519 at 6/12/17 9:10 PM:
--
Thanks [~jzhuge] for the catch! The patch LGTM. One minor thing though, I think
the patch slightly changed the syntax of this function. (Please correct me if
I'm wrong though).
Let's denote condition {{calls.isEmpty() && !shouldCloseConnection.get() &&
running.get()}} as condition A. The original logic works that, it only checks A
before calling wait, meaning when it gets notified, it returns from wait and
just proceed, potentially possible that A is still true at this point. While
the patch changes it that, even when it gets notified, as long as A is true, it
will keep looping. Could you please verify that this change to the syntax is
correct? Namely, {{calls.isEmpty() && !shouldCloseConnection.get() &&
running.get()}} is indeed expected to be false always after the object gets
notified. I'm inclined to believe it is all good based the fact of no test
failures.
was (Author: vagarychen):
Thanks [~jzhuge] for the catch! The patch LGTM. One minor thing though, I think
the patch slightly changed the syntax of this function. (Please correct me if
I'm wrong though).
Let's denote condition {{calls.isEmpty() && !shouldCloseConnection.get() &&
running.get()}} as condition A. The original logic works that, it only checks A
before calling wait, meaning when it gets notified, it returns from wait and
just proceed, potentially possible that A is still true at this point. While
the patch changes it that, even when it gets notified, as long as A is true, it
will keep looping. Could you please verify that this change to the syntax is
correct? I'm inclined to believe it is all good based the fact of no test
failures.
> Client$Connection#waitForWork may suffer spurious wakeup
>
>
> Key: HADOOP-14519
> URL: https://issues.apache.org/jira/browse/HADOOP-14519
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Critical
> Attachments: HADOOP-14519.001.patch
>
>
> {{Client$Connection#waitForWork}} may suffer spurious wakeup because the
> {{wait}} is not surrounded by a loop. See
> [https://docs.oracle.com/javase/7/docs/api/java/lang/Object.html#wait()].
> {code:title=Client$Connection#waitForWork}
> if (calls.isEmpty() && !shouldCloseConnection.get() && running.get()) {
> long timeout = maxIdleTime-
> (Time.now()-lastActivity.get());
> if (timeout>0) {
> try {
> wait(timeout); << spurious wakeup
> } catch (InterruptedException e) {}
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14520) Block compaction for WASB
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Georgi Chalakov updated HADOOP-14520: - Status: Patch Available (was: Open) > Block compaction for WASB > - > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Attachments: HADOOP-14520-01.patch, HADOOP-14520-01-test.txt > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above a predefined, configurable value, > next hflush/hsync triggers the block compaction process. Block compaction > replaces a sequence of blocks with one block. From all the sequences with > total length less than 4M, compaction chooses the longest one. It is a greedy > algorithm that preserve all potential candidates for the next round. Block > Compaction for WASB increases data durability and allows using block blobs > instead of page blobs. By default, block compaction is disabled. Similar to > the configuration for page blobs, the client needs to specify HDFS folders > where block compaction over block blobs is enabled. > Results for HADOOP-14520-01.patch > Tests run: 704, Failures: 0, Errors: 0, Skipped: 119 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14520) Block compaction for WASB
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Georgi Chalakov updated HADOOP-14520: - Status: Open (was: Patch Available) > Block compaction for WASB > - > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Attachments: HADOOP-14520-01.patch, HADOOP-14520-01-test.txt > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above a predefined, configurable value, > next hflush/hsync triggers the block compaction process. Block compaction > replaces a sequence of blocks with one block. From all the sequences with > total length less than 4M, compaction chooses the longest one. It is a greedy > algorithm that preserve all potential candidates for the next round. Block > Compaction for WASB increases data durability and allows using block blobs > instead of page blobs. By default, block compaction is disabled. Similar to > the configuration for page blobs, the client needs to specify HDFS folders > where block compaction over block blobs is enabled. > Results for HADOOP-14520-01.patch > Tests run: 704, Failures: 0, Errors: 0, Skipped: 119 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14519) Client$Connection#waitForWork may suffer spurious wakeup
[
https://issues.apache.org/jira/browse/HADOOP-14519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047047#comment-16047047
]
Chen Liang commented on HADOOP-14519:
-
Thanks [~jzhuge] for the catch! The patch LGTM. One minor thing though, I think
the patch slightly changed the syntax of this function. (Please correct me if
I'm wrong though).
Let's denote condition {{calls.isEmpty() && !shouldCloseConnection.get() &&
running.get()}} as condition A. The original logic works that, it only checks A
before calling wait, meaning when it gets notified, it returns from wait and
just proceed, potentially possible that A is still true at this point. While
the patch changes it that, even when it gets notified, as long as A is true, it
will keep looping. Could you please verify that this change to the syntax is
correct? I'm inclined to believe it is all good based the fact of no test
failures.
> Client$Connection#waitForWork may suffer spurious wakeup
>
>
> Key: HADOOP-14519
> URL: https://issues.apache.org/jira/browse/HADOOP-14519
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Critical
> Attachments: HADOOP-14519.001.patch
>
>
> {{Client$Connection#waitForWork}} may suffer spurious wakeup because the
> {{wait}} is not surrounded by a loop. See
> [https://docs.oracle.com/javase/7/docs/api/java/lang/Object.html#wait()].
> {code:title=Client$Connection#waitForWork}
> if (calls.isEmpty() && !shouldCloseConnection.get() && running.get()) {
> long timeout = maxIdleTime-
> (Time.now()-lastActivity.get());
> if (timeout>0) {
> try {
> wait(timeout); << spurious wakeup
> } catch (InterruptedException e) {}
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14521) KMS client needs retry logic
[
https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046980#comment-16046980
]
Hadoop QA commented on HADOOP-14521:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
4s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 6m
49s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
18s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 19
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 13m 4s{color}
| {color:red} root generated 501 new + 286 unchanged - 0 fixed = 787 total (was
286) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 58s{color} | {color:orange} root: The patch generated 2 new + 137 unchanged
- 0 fixed = 139 total (was 137) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
59s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 10s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}147m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestAclsEndToEnd |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14521 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872745/HDFS-11804-trunk-6.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux b42f19aae496 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 86368cc |
| Default Java | 1.8.0_131 |
| compile |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12521/artifact/patchprocess/branch-compile-root.txt
|
| findbugs | v3.1.0-RC1 |
| findbugs |
[jira] [Commented] (HADOOP-13174) Add more debug logs for delegation tokens and authentication
[ https://issues.apache.org/jira/browse/HADOOP-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046969#comment-16046969 ] Xiao Chen commented on HADOOP-13174: Clarified with [~yzhangal] offline about his question: bq. provide a set of debug configuration setting here so we can refer to when needed? >From my experience, it's usually easier if we change general log levels as the >comment above. If that's not acceptable for the environment, what we need are >(from my experience): For server: - org.apache.hadoop.security.authentication.server.AuthenticationFilter - org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler - org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter - org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler - org.apache.hadoop.crypto.key.kms.server.* For client: - org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator - org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL - org.apache.hadoop.security.authentication.client.KerberosAuthenticator - org.apache.hadoop.security.authentication.client.AuthenticatedURL - org.apache.hadoop.crypto.key.kms.* > Add more debug logs for delegation tokens and authentication > > > Key: HADOOP-13174 > URL: https://issues.apache.org/jira/browse/HADOOP-13174 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: HADOOP-13174.01.patch, HADOOP-13174.02.patch, > HADOOP-13174.03.patch, HADOOP-13174.04.patch, HADOOP-13174.05.patch > > > Recently I debugged several authentication related problems, and found that > the debug logs are not enough to identify a problem. > This jira improves it by adding more debug/trace logs along the line. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14503) Make RollingAverages a mutable metric
[ https://issues.apache.org/jira/browse/HADOOP-14503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HADOOP-14503: Attachment: HADOOP-14503.006.patch > Make RollingAverages a mutable metric > - > > Key: HADOOP-14503 > URL: https://issues.apache.org/jira/browse/HADOOP-14503 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru > Attachments: HADOOP-14503.001.patch, HADOOP-14503.002.patch, > HADOOP-14503.003.patch, HADOOP-14503.004.patch, HADOOP-14503.005.patch, > HADOOP-14503.006.patch > > > RollingAverages metric extends on MutableRatesWithAggregation metric and > maintains a group of rolling average metrics. This class should be allowed to > register as a metric with the MetricSystem. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14497) Logs for KMS delegation token lifecycle
[ https://issues.apache.org/jira/browse/HADOOP-14497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046922#comment-16046922 ] Yongjun Zhang commented on HADOOP-14497: Thanks [~xiaochen]! > Logs for KMS delegation token lifecycle > --- > > Key: HADOOP-14497 > URL: https://issues.apache.org/jira/browse/HADOOP-14497 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Yongjun Zhang >Assignee: Xiao Chen > Fix For: 2.9.0, 3.0.0-alpha4 > > > We run into quite some customer cases about authentication failures related > to KMS delegation token. It would be nice to see a log for each stage of the > token: > 1. creation > 2. renewal > 3. removal upon cancel > 4. remove upon expiration > So that when we correlate the logs for the same DT, we can have a good > picture about what's going on, and what could have caused the authentication > failure. > The same is applicable to other delegation tokens. > NOTE: When log info about delagation token, we don't want leak user's secret > info. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14429) FTPFileSystem#getFsAction always returns FsAction.NONE
[
https://issues.apache.org/jira/browse/HADOOP-14429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046831#comment-16046831
]
Yongjun Zhang commented on HADOOP-14429:
Thanks [~Hongyuan Li]. +1 after one nit: it would have better readability if we
have a empty line added between different sections in the unit test. Sorry I
did not look at this earlier.
> FTPFileSystem#getFsAction always returns FsAction.NONE
> ---
>
> Key: HADOOP-14429
> URL: https://issues.apache.org/jira/browse/HADOOP-14429
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0-alpha2
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
> Attachments: HADOOP-14429-001.patch, HADOOP-14429-002.patch,
> HADOOP-14429-003.patch, HADOOP-14429-004.patch
>
>
>
> {code}
> private FsAction getFsAction(int accessGroup, FTPFile ftpFile) {
> FsAction action = FsAction.NONE;
> if (ftpFile.hasPermission(accessGroup, FTPFile.READ_PERMISSION)) {
> action.or(FsAction.READ);
> }
> if (ftpFile.hasPermission(accessGroup, FTPFile.WRITE_PERMISSION)) {
> action.or(FsAction.WRITE);
> }
> if (ftpFile.hasPermission(accessGroup, FTPFile.EXECUTE_PERMISSION)) {
> action.or(FsAction.EXECUTE);
> }
> return action;
> }
> {code}
> from code above, we can see that the getFsAction method doesnot modify the
> action generated by FsAction action = FsAction.NONE,which means it return
> FsAction.NONE all the time;
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14310) RolloverSignerSecretProvider.LOG should be @VisibleForTesting
[ https://issues.apache.org/jira/browse/HADOOP-14310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046805#comment-16046805 ] Hudson commented on HADOOP-14310: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11857 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11857/]) HADOOP-14310. RolloverSignerSecretProvider.LOG should be (templedf: rev 86368cc7667adfe224df77b039c2ffe8de7f889a) * (edit) hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/RolloverSignerSecretProvider.java * (edit) hadoop-common-project/hadoop-auth/pom.xml > RolloverSignerSecretProvider.LOG should be @VisibleForTesting > - > > Key: HADOOP-14310 > URL: https://issues.apache.org/jira/browse/HADOOP-14310 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Arun Shanmugam Kumar >Priority: Minor > Labels: newbie > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: HADOOP-14310.001.patch, HADOOP-14310.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-14521) KMS client needs retry logic
[ https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah moved HDFS-11804 to HADOOP-14521: Affects Version/s: (was: 2.6.0) 2.6.0 Target Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2 (was: 3.0.0-alpha4, 2.8.2) Issue Type: Improvement (was: Bug) Key: HADOOP-14521 (was: HDFS-11804) Project: Hadoop Common (was: Hadoop HDFS) > KMS client needs retry logic > > > Key: HADOOP-14521 > URL: https://issues.apache.org/jira/browse/HADOOP-14521 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch, > HDFS-11804-trunk-3.patch, HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch, > HDFS-11804-trunk-6.patch, HDFS-11804-trunk.patch > > > The kms client appears to have no retry logic – at all. It's completely > decoupled from the ipc retry logic. This has major impacts if the KMS is > unreachable for any reason, including but not limited to network connection > issues, timeouts, the +restart during an upgrade+. > This has some major ramifications: > # Jobs may fail to submit, although oozie resubmit logic should mask it > # Non-oozie launchers may experience higher rates if they do not already have > retry logic. > # Tasks reading EZ files will fail, probably be masked by framework reattempts > # EZ file creation fails after creating a 0-length file – client receives > EDEK in the create response, then fails when decrypting the EDEK > # Bulk hadoop fs copies, and maybe distcp, will prematurely fail -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14521) KMS client needs retry logic
[ https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046793#comment-16046793 ] Rushabh S Shah commented on HADOOP-14521: - bq. Should this JIRA be moved to project "Hadoop Common" since KMS belongs to common? Moved to common. Thanks [~jzhuge]. > KMS client needs retry logic > > > Key: HADOOP-14521 > URL: https://issues.apache.org/jira/browse/HADOOP-14521 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch, > HDFS-11804-trunk-3.patch, HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch, > HDFS-11804-trunk-6.patch, HDFS-11804-trunk.patch > > > The kms client appears to have no retry logic – at all. It's completely > decoupled from the ipc retry logic. This has major impacts if the KMS is > unreachable for any reason, including but not limited to network connection > issues, timeouts, the +restart during an upgrade+. > This has some major ramifications: > # Jobs may fail to submit, although oozie resubmit logic should mask it > # Non-oozie launchers may experience higher rates if they do not already have > retry logic. > # Tasks reading EZ files will fail, probably be masked by framework reattempts > # EZ file creation fails after creating a 0-length file – client receives > EDEK in the create response, then fails when decrypting the EDEK > # Bulk hadoop fs copies, and maybe distcp, will prematurely fail -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14310) RolloverSignerSecretProvider.LOG should be @VisibleForTesting
[ https://issues.apache.org/jira/browse/HADOOP-14310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HADOOP-14310: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha4 2.9.0 Status: Resolved (was: Patch Available) Thanks for the patch, [~arunsk]. Committed to trunk and branch-2. > RolloverSignerSecretProvider.LOG should be @VisibleForTesting > - > > Key: HADOOP-14310 > URL: https://issues.apache.org/jira/browse/HADOOP-14310 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Arun Shanmugam Kumar >Priority: Minor > Labels: newbie > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: HADOOP-14310.001.patch, HADOOP-14310.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14440) Add metrics for connections dropped
[ https://issues.apache.org/jira/browse/HADOOP-14440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HADOOP-14440: --- Fix Version/s: 2.7.4 Thanks for the work [~ebadger]. I think this is a good improvement for 2.7.4; just backported to branch-2.7. > Add metrics for connections dropped > --- > > Key: HADOOP-14440 > URL: https://issues.apache.org/jira/browse/HADOOP-14440 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Eric Badger >Assignee: Eric Badger > Fix For: 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2 > > Attachments: HADOOP-14440.001.patch, HADOOP-14440.002.patch, > HADOOP-14440.003.patch > > > Will be useful to figure out when the NN is getting overloaded with more > connections than it can handle -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14443) Azure: Add retry and client side failover for authorization, SASKey generation and delegation token generation requests to remote service
[
https://issues.apache.org/jira/browse/HADOOP-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046646#comment-16046646
]
Hadoop QA commented on HADOOP-14443:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} hadoop-tools/hadoop-azure: The patch generated 0 new
+ 27 unchanged - 23 fixed = 27 total (was 50) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
18s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14443 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872709/HADOOP-14443.4.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c4ffdac51068 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d64c842 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12520/testReport/ |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12520/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Azure: Add retry and client side failover for authorization, SASKey
> generation and delegation token generation requests to remote service
> -
>
> Key: HADOOP-14443
> URL: https://issues.apache.org/jira/browse/HADOOP-14443
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-14443.1.patch, HADOOP-14443.2.patch,
> HADOOP-14443.3.patch, HADOOP-14443.4.patch
>
>
> Currently, {{WasRemoteCallHelper}} can be configured
[jira] [Updated] (HADOOP-14443) Azure: Add retry and client side failover for authorization, SASKey generation and delegation token generation requests to remote service
[
https://issues.apache.org/jira/browse/HADOOP-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14443:
--
Attachment: HADOOP-14443.4.patch
Attaching {{HADOOP-14443.4.patch}} to fix {{javac}}, {{Javadoc}} subsystem
related failures.
> Azure: Add retry and client side failover for authorization, SASKey
> generation and delegation token generation requests to remote service
> -
>
> Key: HADOOP-14443
> URL: https://issues.apache.org/jira/browse/HADOOP-14443
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-14443.1.patch, HADOOP-14443.2.patch,
> HADOOP-14443.3.patch, HADOOP-14443.4.patch
>
>
> Currently, {{WasRemoteCallHelper}} can be configured to talk to only one URL
> for authorization, SASKey generation and delegation token generation. If for
> some reason the service is down, all the requests will fail.
> So proposal is to,
> - Add support to configure multiple URLs, so that if communication to one URL
> fails, client can retry on another instance of the service running on
> different node for authorization, SASKey generation and delegation token
> generation.
> - Rename the configurations {{fs.azure.authorization.remote.service.url}} to
> {{fs.azure.authorization.remote.service.urls}} and
> {{fs.azure.cred.service.url}} to {{fs.azure.cred.service.urls}} to support
> the comma separated list of URLs.
> Introduce a new configuration {{fs.azure.delegation.token.service.urls}} to
> configure the comma separated list of service URLs to get the delegation
> token.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14443) Azure: Add retry and client side failover for authorization, SASKey generation and delegation token generation requests to remote service
[
https://issues.apache.org/jira/browse/HADOOP-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046411#comment-16046411
]
Hadoop QA commented on HADOOP-14443:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 18s{color}
| {color:red} hadoop-tools_hadoop-azure generated 1 new + 1 unchanged - 0 fixed
= 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} hadoop-tools/hadoop-azure: The patch generated 0 new
+ 27 unchanged - 23 fixed = 27 total (was 50) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
12s{color} | {color:red} hadoop-azure in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
27s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14443 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872671/HADOOP-14443.3.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d2e30b1c08b3 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e86eef9 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12519/artifact/patchprocess/diff-compile-javac-hadoop-tools_hadoop-azure.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12519/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12519/testReport/ |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12519/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Azure: Add retry and client side failover for authorization, SASKey
> generation and delegation token generation requests to remote service
> -
>
> Key: HADOOP-14443
> URL: https://issues.apache.org/jira/browse/HADOOP-14443
> Project: Hadoop Common
> Issue Type: Improvement
>
[jira] [Updated] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, which will cause recursively list stuck
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongyuan Li updated HADOOP-14469:
-
Summary: FTPFileSystem#listStatus get currentPath and parentPath at the
same time, which will cause recursively list stuck (was:
FTPFileSystem#listStatus get currentPath and parentPath, which will cause
recursively list stuck)
> FTPFileSystem#listStatus get currentPath and parentPath at the same time,
> which will cause recursively list stuck
> -
>
> Key: HADOOP-14469
> URL: https://issues.apache.org/jira/browse/HADOOP-14469
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0-alpha3
> Environment: ftp build by windows7 + Serv-U_64 12.1.0.8
> code runs any os
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Critical
> Attachments: HADOOP-14469-001.patch, HADOOP-14469-002.patch,
> HADOOP-14469-003.patch
>
>
> for some ftpsystems, liststatus method will return new Path(".") and new
> Path(".."), thus causing list op looping.for example, Serv-U
> We can see the logic in code below:
> {code}
> private FileStatus[] listStatus(FTPClient client, Path file)
> throws IOException {
> ……
> FileStatus[] fileStats = new FileStatus[ftpFiles.length];
> for (int i = 0; i < ftpFiles.length; i++) {
> fileStats[i] = getFileStatus(ftpFiles[i], absolute);
> }
> return fileStats;
> }
> {code}
> {code}
> public void test() throws Exception{
> FTPFileSystem ftpFileSystem = new FTPFileSystem();
> ftpFileSystem.initialize(new
> Path("ftp://test:123456@192.168.44.1/;).toUri(),
> new Configuration());
> FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
> for(FileStatus fileStatus1 : fileStatus)
> System.out.println(fileStatus1);
> }
> {code}
> using test code below, the test results list below
> {code}
> FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
> isDirectory=false; length=2036; replication=1; blocksize=4096;
> modification_time=149579778; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
> isDirectory=false; length=1322; replication=1; blocksize=4096;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
> isDirectory=true; modification_time=149579712; access_time=0; owner=user;
> group=group; permission=-; isSymlink=false}
> {code}
> In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}}
> is obviously the current Path, and
> {{FileStatus{path=ftp://test:123456@192.168.44.1/;…… }} is obviously the
> parent Path.
> So, if we want to walk the directory recursively, it will stuck.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14429) FTPFileSystem#getFsAction always returns FsAction.NONE
[
https://issues.apache.org/jira/browse/HADOOP-14429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongyuan Li updated HADOOP-14429:
-
Priority: Major (was: Trivial)
> FTPFileSystem#getFsAction always returns FsAction.NONE
> ---
>
> Key: HADOOP-14429
> URL: https://issues.apache.org/jira/browse/HADOOP-14429
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0-alpha2
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
> Attachments: HADOOP-14429-001.patch, HADOOP-14429-002.patch,
> HADOOP-14429-003.patch, HADOOP-14429-004.patch
>
>
>
> {code}
> private FsAction getFsAction(int accessGroup, FTPFile ftpFile) {
> FsAction action = FsAction.NONE;
> if (ftpFile.hasPermission(accessGroup, FTPFile.READ_PERMISSION)) {
> action.or(FsAction.READ);
> }
> if (ftpFile.hasPermission(accessGroup, FTPFile.WRITE_PERMISSION)) {
> action.or(FsAction.WRITE);
> }
> if (ftpFile.hasPermission(accessGroup, FTPFile.EXECUTE_PERMISSION)) {
> action.or(FsAction.EXECUTE);
> }
> return action;
> }
> {code}
> from code above, we can see that the getFsAction method doesnot modify the
> action generated by FsAction action = FsAction.NONE,which means it return
> FsAction.NONE all the time;
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14422) FTPFileSystem and distcp does not work when ecnountered a complex ftp password with char ':' or '@'
[
https://issues.apache.org/jira/browse/HADOOP-14422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046398#comment-16046398
]
Hongyuan Li commented on HADOOP-14422:
--
Ok, you can close the issue.
> FTPFileSystem and distcp does not work when ecnountered a complex ftp
> password with char ':' or '@'
> ---
>
> Key: HADOOP-14422
> URL: https://issues.apache.org/jira/browse/HADOOP-14422
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, tools/distcp
>Affects Versions: 3.0.0-alpha2
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
> Attachments: HADOOP-14422-002.patch, HADOOP-14422-003.patch,
> HADOOP-14422-004.patch, HADOOP-14422-005.patch, HADOOP-14422.patch
>
>
> ftp username: test passwd:1@123
> hadoop distcp ftp://test:1@123@node2/home/test/ hdfs://piaopiao/test
> 17/05/15 15:24:56 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
> copyStrategy='uniformsize', sourceFileListing=null,
> sourcePaths=[ftp://test:1@123@node2/home/test], targetPath=/test,
> targetPathExists=true, preserveRawXattrs=false, filtersFile='null'}
> 17/05/15 15:24:57 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
> to rm2
> 17/05/15 15:24:59 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Login failed on server - 0.0.0.0, port - 21 as user
> 'null'
> at
> org.apache.hadoop.fs.ftp.FTPFileSystem.connect(FTPFileSystem.java:154)
> at
> org.apache.hadoop.fs.ftp.FTPFileSystem.getFileStatus(FTPFileSystem.java:415)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:68)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:263)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1630)
> at
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
> at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
> at
> org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:389)
> at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:187)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:153)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:126)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:453)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath, which will cause recursively list stuck
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongyuan Li updated HADOOP-14469:
-
Priority: Critical (was: Major)
> FTPFileSystem#listStatus get currentPath and parentPath, which will cause
> recursively list stuck
>
>
> Key: HADOOP-14469
> URL: https://issues.apache.org/jira/browse/HADOOP-14469
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0-alpha3
> Environment: ftp build by windows7 + Serv-U_64 12.1.0.8
> code runs any os
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Critical
> Attachments: HADOOP-14469-001.patch, HADOOP-14469-002.patch,
> HADOOP-14469-003.patch
>
>
> for some ftpsystems, liststatus method will return new Path(".") and new
> Path(".."), thus causing list op looping.for example, Serv-U
> We can see the logic in code below:
> {code}
> private FileStatus[] listStatus(FTPClient client, Path file)
> throws IOException {
> ……
> FileStatus[] fileStats = new FileStatus[ftpFiles.length];
> for (int i = 0; i < ftpFiles.length; i++) {
> fileStats[i] = getFileStatus(ftpFiles[i], absolute);
> }
> return fileStats;
> }
> {code}
> {code}
> public void test() throws Exception{
> FTPFileSystem ftpFileSystem = new FTPFileSystem();
> ftpFileSystem.initialize(new
> Path("ftp://test:123456@192.168.44.1/;).toUri(),
> new Configuration());
> FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
> for(FileStatus fileStatus1 : fileStatus)
> System.out.println(fileStatus1);
> }
> {code}
> using test code below, the test results list below
> {code}
> FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
> isDirectory=false; length=2036; replication=1; blocksize=4096;
> modification_time=149579778; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
> isDirectory=false; length=1322; replication=1; blocksize=4096;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
> isDirectory=true; modification_time=149579712; access_time=0; owner=user;
> group=group; permission=-; isSymlink=false}
> {code}
> In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}}
> is obviously the current Path, and
> {{FileStatus{path=ftp://test:123456@192.168.44.1/;…… }} is obviously the
> parent Path.
> So, if we want to walk the directory recursively, it will stuck.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11886) Failed to run distcp against ftp server installed on Windows
[ https://issues.apache.org/jira/browse/HADOOP-11886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046394#comment-16046394 ] Hongyuan Li commented on HADOOP-11886: -- On windows os ,you could set fs.ftp.transfer.mode to FTP.STREAM_TRANSFER_MODE which is equal to 10. > Failed to run distcp against ftp server installed on Windows > > > Key: HADOOP-11886 > URL: https://issues.apache.org/jira/browse/HADOOP-11886 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp >Reporter: sam liu >Assignee: sam liu >Priority: Blocker > > Could run distcp against ftp server installed on Linux, but could NOT run > distcp against ftp server installed on Windows(such as IIS ftp service). > However, distcp works well for FileZilla ftp server installed on Windows -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14443) Azure: Add retry and client side failover for authorization, SASKey generation and delegation token generation requests to remote service
[
https://issues.apache.org/jira/browse/HADOOP-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14443:
--
Attachment: HADOOP-14443.3.patch
[~liuml07] Thanks for the comments. I have fixed few of those and attached a
new patch,
1. {{@Override}} annotation to new line.
2. Marked the parameters such as {{op}}, {{renewer}}, {{service}} and
{{token}} as constants?
3. There is a default retry spec which will be used, if user wants to override,
he can do so. If this is not recommended in Hadoop codebase, we can remove the
configuration.
4. Added {{Thread.currentThread().interrupt()}} in
{{WasbRemoteCallHelper::shouldRetry()}} when {{InterruptedException}} occurs.
5. This patch consolidates various http helpers which make common SPNEGO
connection into {{WasbRemoteCallHelper}} and {{SecureWasbRemoteCallHelper}}. It
makes easier to apply the same retry/failover logic to various remote HTTP
calls. I am afraid separating it may be difficult.
> Azure: Add retry and client side failover for authorization, SASKey
> generation and delegation token generation requests to remote service
> -
>
> Key: HADOOP-14443
> URL: https://issues.apache.org/jira/browse/HADOOP-14443
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-14443.1.patch, HADOOP-14443.2.patch,
> HADOOP-14443.3.patch
>
>
> Currently, {{WasRemoteCallHelper}} can be configured to talk to only one URL
> for authorization, SASKey generation and delegation token generation. If for
> some reason the service is down, all the requests will fail.
> So proposal is to,
> - Add support to configure multiple URLs, so that if communication to one URL
> fails, client can retry on another instance of the service running on
> different node for authorization, SASKey generation and delegation token
> generation.
> - Rename the configurations {{fs.azure.authorization.remote.service.url}} to
> {{fs.azure.authorization.remote.service.urls}} and
> {{fs.azure.cred.service.url}} to {{fs.azure.cred.service.urls}} to support
> the comma separated list of URLs.
> Introduce a new configuration {{fs.azure.delegation.token.service.urls}} to
> configure the comma separated list of service URLs to get the delegation
> token.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
