[jira] [Commented] (HDFS-9887) WebHdfs socket timeouts should be configurable
[ https://issues.apache.org/jira/browse/HDFS-9887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183322#comment-15183322 ] Xiaoyu Yao commented on HDFS-9887: -- Filed HDFS-9914 for the fix. > WebHdfs socket timeouts should be configurable > -- > > Key: HDFS-9887 > URL: https://issues.apache.org/jira/browse/HDFS-9887 > Project: Hadoop HDFS > Issue Type: Improvement > Components: fs, webhdfs > Environment: all >Reporter: Austin Donnelly >Assignee: Austin Donnelly > Labels: easyfix, newbie > Fix For: 2.8.0 > > Attachments: HADOOP-12827.001.patch, HADOOP-12827.002.patch, > HADOOP-12827.002.patch, HADOOP-12827.002.patch, HADOOP-12827.003.patch, > HADOOP-12827.004.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > WebHdfs client connections use sockets with fixed timeouts of 60 seconds to > connect, and 60 seconds for reads. > This is a problem because I am trying to use WebHdfs to access an archive > storage system which can take minutes to hours to return the requested data > over WebHdfs. > The fix is to add new configuration file options to allow these 60s defaults > to be customised in hdfs-site.xml. > If the new configuration options are not present, the behavior is unchanged > from before. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9887) WebHdfs socket timeouts should be configurable
[
https://issues.apache.org/jira/browse/HDFS-9887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183301#comment-15183301
]
Xiaoyu Yao commented on HDFS-9887:
--
Thanks [~jojochuang] for reporting this. Further reading found that the webhdfs
specific read/connect timeout implemented by HDFS-9887 should not affect other
callers of {{URLConnectionFactory.newSslConnConfigurator()}} such as
{{QuorumJournalManager#QuorumJournalManger(), DFSck#DFSck() and
TransferFsImage()}}. I will file separate ticket to fix it.
> WebHdfs socket timeouts should be configurable
> --
>
> Key: HDFS-9887
> URL: https://issues.apache.org/jira/browse/HDFS-9887
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: fs, webhdfs
> Environment: all
>Reporter: Austin Donnelly
>Assignee: Austin Donnelly
> Labels: easyfix, newbie
> Fix For: 2.8.0
>
> Attachments: HADOOP-12827.001.patch, HADOOP-12827.002.patch,
> HADOOP-12827.002.patch, HADOOP-12827.002.patch, HADOOP-12827.003.patch,
> HADOOP-12827.004.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> WebHdfs client connections use sockets with fixed timeouts of 60 seconds to
> connect, and 60 seconds for reads.
> This is a problem because I am trying to use WebHdfs to access an archive
> storage system which can take minutes to hours to return the requested data
> over WebHdfs.
> The fix is to add new configuration file options to allow these 60s defaults
> to be customised in hdfs-site.xml.
> If the new configuration options are not present, the behavior is unchanged
> from before.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9887) WebHdfs socket timeouts should be configurable
[
https://issues.apache.org/jira/browse/HDFS-9887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183241#comment-15183241
]
Xiaoyu Yao commented on HDFS-9887:
--
Agree, this is a bug. Webhdfs with ssl configuration exception will not honor
the configurable webhdfs connect/read timeout. It will always be
{{DEFAULT_TIMEOUT_CONN_CONFIGURATOR}} the default value (1 min).
> WebHdfs socket timeouts should be configurable
> --
>
> Key: HDFS-9887
> URL: https://issues.apache.org/jira/browse/HDFS-9887
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: fs, webhdfs
> Environment: all
>Reporter: Austin Donnelly
>Assignee: Austin Donnelly
> Labels: easyfix, newbie
> Fix For: 2.8.0
>
> Attachments: HADOOP-12827.001.patch, HADOOP-12827.002.patch,
> HADOOP-12827.002.patch, HADOOP-12827.002.patch, HADOOP-12827.003.patch,
> HADOOP-12827.004.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> WebHdfs client connections use sockets with fixed timeouts of 60 seconds to
> connect, and 60 seconds for reads.
> This is a problem because I am trying to use WebHdfs to access an archive
> storage system which can take minutes to hours to return the requested data
> over WebHdfs.
> The fix is to add new configuration file options to allow these 60s defaults
> to be customised in hdfs-site.xml.
> If the new configuration options are not present, the behavior is unchanged
> from before.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9887) WebHdfs socket timeouts should be configurable
[ https://issues.apache.org/jira/browse/HDFS-9887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183218#comment-15183218 ] Wei-Chiu Chuang commented on HDFS-9887: --- I think this patch is not complete -- if for some reason SSL configuration throws an exception, the socket timeouts will not be configured, even if the connection is supposed to be webhdfs only, not swebhdfs. > WebHdfs socket timeouts should be configurable > -- > > Key: HDFS-9887 > URL: https://issues.apache.org/jira/browse/HDFS-9887 > Project: Hadoop HDFS > Issue Type: Improvement > Components: fs, webhdfs > Environment: all >Reporter: Austin Donnelly >Assignee: Austin Donnelly > Labels: easyfix, newbie > Fix For: 2.8.0 > > Attachments: HADOOP-12827.001.patch, HADOOP-12827.002.patch, > HADOOP-12827.002.patch, HADOOP-12827.002.patch, HADOOP-12827.003.patch, > HADOOP-12827.004.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > WebHdfs client connections use sockets with fixed timeouts of 60 seconds to > connect, and 60 seconds for reads. > This is a problem because I am trying to use WebHdfs to access an archive > storage system which can take minutes to hours to return the requested data > over WebHdfs. > The fix is to add new configuration file options to allow these 60s defaults > to be customised in hdfs-site.xml. > If the new configuration options are not present, the behavior is unchanged > from before. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9887) WebHdfs socket timeouts should be configurable
[
https://issues.apache.org/jira/browse/HDFS-9887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176389#comment-15176389
]
Hadoop QA commented on HDFS-9887:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
48s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 15s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 49s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 67m 50s
{color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 33s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {colo
[jira] [Commented] (HDFS-9887) WebHdfs socket timeouts should be configurable
[ https://issues.apache.org/jira/browse/HDFS-9887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176047#comment-15176047 ] Hudson commented on HDFS-9887: -- FAILURE: Integrated in Hadoop-trunk-Commit #9411 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9411/]) HDFS-9887. WebHdfs socket timeouts should be configurable. Contributed (xyao: rev 5abf051249d485313dfffc6aeff6f81c0da1f623) * hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTimeouts.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/URLConnectionFactory.java > WebHdfs socket timeouts should be configurable > -- > > Key: HDFS-9887 > URL: https://issues.apache.org/jira/browse/HDFS-9887 > Project: Hadoop HDFS > Issue Type: Improvement > Components: fs, webhdfs > Environment: all >Reporter: Austin Donnelly >Assignee: Austin Donnelly > Labels: easyfix, newbie > Fix For: 2.8.0 > > Attachments: HADOOP-12827.001.patch, HADOOP-12827.002.patch, > HADOOP-12827.002.patch, HADOOP-12827.002.patch, HADOOP-12827.003.patch, > HADOOP-12827.004.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > WebHdfs client connections use sockets with fixed timeouts of 60 seconds to > connect, and 60 seconds for reads. > This is a problem because I am trying to use WebHdfs to access an archive > storage system which can take minutes to hours to return the requested data > over WebHdfs. > The fix is to add new configuration file options to allow these 60s defaults > to be customised in hdfs-site.xml. > If the new configuration options are not present, the behavior is unchanged > from before. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9887) WebHdfs socket timeouts should be configurable
[ https://issues.apache.org/jira/browse/HDFS-9887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176040#comment-15176040 ] Austin Donnelly commented on HDFS-9887: --- Thank you! > WebHdfs socket timeouts should be configurable > -- > > Key: HDFS-9887 > URL: https://issues.apache.org/jira/browse/HDFS-9887 > Project: Hadoop HDFS > Issue Type: Improvement > Components: fs, webhdfs > Environment: all >Reporter: Austin Donnelly >Assignee: Austin Donnelly > Labels: easyfix, newbie > Fix For: 2.8.0 > > Attachments: HADOOP-12827.001.patch, HADOOP-12827.002.patch, > HADOOP-12827.002.patch, HADOOP-12827.002.patch, HADOOP-12827.003.patch, > HADOOP-12827.004.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > WebHdfs client connections use sockets with fixed timeouts of 60 seconds to > connect, and 60 seconds for reads. > This is a problem because I am trying to use WebHdfs to access an archive > storage system which can take minutes to hours to return the requested data > over WebHdfs. > The fix is to add new configuration file options to allow these 60s defaults > to be customised in hdfs-site.xml. > If the new configuration options are not present, the behavior is unchanged > from before. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
