[jira] [Commented] (HDFS-13949) Correct the description of dfs.datanode.disk.check.timeout in hdfs-default.xml

2018-10-10 Thread Toshihiro Suzuki (JIRA) 


[ 
https://issues.apache.org/jira/browse/HDFS-13949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644571#comment-16644571
 ] 

Toshihiro Suzuki commented on HDFS-13949:
-

Thank you for reviewing and committing the patch. [~nandakumar131]

> Correct the description of dfs.datanode.disk.check.timeout in hdfs-default.xml
> --
>
> Key: HDFS-13949
> URL: https://issues.apache.org/jira/browse/HDFS-13949
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HDFS-13949.1.patch
>
>
> The description of dfs.datanode.disk.check.timeout in hdfs-default.xml is as 
> follows:
> {code}
> 
>   dfs.datanode.disk.check.timeout
>   10m
>   
> Maximum allowed time for a disk check to complete during DataNode
> startup. If the check does not complete within this time interval
> then the disk is declared as failed. This setting supports
> multiple time unit suffixes as described in dfs.heartbeat.interval.
> If no suffix is specified then milliseconds is assumed.
>   
> 
> {code}
> I don't think the value of this config is used only during DataNode startup. 
> I think it's used whenever checking volumes.
> The description is misleading so we need to correct it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-13949) Correct the description of dfs.datanode.disk.check.timeout in hdfs-default.xml

2018-10-10 Thread Hudson (JIRA) 


[ 
https://issues.apache.org/jira/browse/HDFS-13949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644554#comment-16644554
 ] 

Hudson commented on HDFS-13949:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15165 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/15165/])
HDFS-13949. Correct the description of dfs.datanode.disk.check.timeout (nanda: 
rev b39b802422b444629095bde4484b820f02fd6d1d)
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml


> Correct the description of dfs.datanode.disk.check.timeout in hdfs-default.xml
> --
>
> Key: HDFS-13949
> URL: https://issues.apache.org/jira/browse/HDFS-13949
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HDFS-13949.1.patch
>
>
> The description of dfs.datanode.disk.check.timeout in hdfs-default.xml is as 
> follows:
> {code}
> 
>   dfs.datanode.disk.check.timeout
>   10m
>   
> Maximum allowed time for a disk check to complete during DataNode
> startup. If the check does not complete within this time interval
> then the disk is declared as failed. This setting supports
> multiple time unit suffixes as described in dfs.heartbeat.interval.
> If no suffix is specified then milliseconds is assumed.
>   
> 
> {code}
> I don't think the value of this config is used only during DataNode startup. 
> I think it's used whenever checking volumes.
> The description is misleading so we need to correct it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-13949) Correct the description of dfs.datanode.disk.check.timeout in hdfs-default.xml

2018-10-10 Thread Nanda kumar (JIRA) 


[ 
https://issues.apache.org/jira/browse/HDFS-13949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644540#comment-16644540
 ] 

Nanda kumar commented on HDFS-13949:


Thanks for the contribution [~brfrn169]. Committed it to trunk.
I have added you as a contributor to HDFS project, from now on you should be 
able to assign HDFS jiras to yourself.

> Correct the description of dfs.datanode.disk.check.timeout in hdfs-default.xml
> --
>
> Key: HDFS-13949
> URL: https://issues.apache.org/jira/browse/HDFS-13949
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Minor
> Attachments: HDFS-13949.1.patch
>
>
> The description of dfs.datanode.disk.check.timeout in hdfs-default.xml is as 
> follows:
> {code}
> 
>   dfs.datanode.disk.check.timeout
>   10m
>   
> Maximum allowed time for a disk check to complete during DataNode
> startup. If the check does not complete within this time interval
> then the disk is declared as failed. This setting supports
> multiple time unit suffixes as described in dfs.heartbeat.interval.
> If no suffix is specified then milliseconds is assumed.
>   
> 
> {code}
> I don't think the value of this config is used only during DataNode startup. 
> I think it's used whenever checking volumes.
> The description is misleading so we need to correct it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-13949) Correct the description of dfs.datanode.disk.check.timeout in hdfs-default.xml

2018-10-10 Thread Nanda kumar (JIRA) 


[ 
https://issues.apache.org/jira/browse/HDFS-13949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644535#comment-16644535
 ] 

Nanda kumar commented on HDFS-13949:


[~brfrn169], thanks for the update. Agreed, the description change looks good.
+1, I will commit this shortly.

> Correct the description of dfs.datanode.disk.check.timeout in hdfs-default.xml
> --
>
> Key: HDFS-13949
> URL: https://issues.apache.org/jira/browse/HDFS-13949
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Minor
> Attachments: HDFS-13949.1.patch
>
>
> The description of dfs.datanode.disk.check.timeout in hdfs-default.xml is as 
> follows:
> {code}
> 
>   dfs.datanode.disk.check.timeout
>   10m
>   
> Maximum allowed time for a disk check to complete during DataNode
> startup. If the check does not complete within this time interval
> then the disk is declared as failed. This setting supports
> multiple time unit suffixes as described in dfs.heartbeat.interval.
> If no suffix is specified then milliseconds is assumed.
>   
> 
> {code}
> I don't think the value of this config is used only during DataNode startup. 
> I think it's used whenever checking volumes.
> The description is misleading so we need to correct it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-13949) Correct the description of dfs.datanode.disk.check.timeout in hdfs-default.xml

2018-10-09 Thread Toshihiro Suzuki (JIRA) 


[ 
https://issues.apache.org/jira/browse/HDFS-13949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644253#comment-16644253
 ] 

Toshihiro Suzuki commented on HDFS-13949:
-

Thank you very much for reviewing [~nandakumar131].

The property has been also used in ThrottledAsyncChecker that's initialized in 
the constructor of DatasetVolumeChecker:
{code}
diskCheckTimeout = conf.getTimeDuration(
DFSConfigKeys.DFS_DATANODE_DISK_CHECK_TIMEOUT_KEY,
DFSConfigKeys.DFS_DATANODE_DISK_CHECK_TIMEOUT_DEFAULT,
TimeUnit.MILLISECONDS);

delegateChecker = new ThrottledAsyncChecker<>(
timer, minDiskCheckGapMs, diskCheckTimeout,
Executors.newCachedThreadPool(
new ThreadFactoryBuilder()
.setNameFormat("DataNode DiskChecker thread %d")
.setDaemon(true)
.build()));
{code}
This timeout is used in ThrottledAsyncChecker#schedule. And this method is 
called by DatasetVolumeChecker#checkVolume. DatasetVolumeChecker#checkVolume is 
called by DataNode#checkDiskErrorAsync that's called when there might possibly 
be a disk failure. So it looks like to me the property is not only used during 
DataNode startup.

> Correct the description of dfs.datanode.disk.check.timeout in hdfs-default.xml
> --
>
> Key: HDFS-13949
> URL: https://issues.apache.org/jira/browse/HDFS-13949
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Minor
> Attachments: HDFS-13949.1.patch
>
>
> The description of dfs.datanode.disk.check.timeout in hdfs-default.xml is as 
> follows:
> {code}
> 
>   dfs.datanode.disk.check.timeout
>   10m
>   
> Maximum allowed time for a disk check to complete during DataNode
> startup. If the check does not complete within this time interval
> then the disk is declared as failed. This setting supports
> multiple time unit suffixes as described in dfs.heartbeat.interval.
> If no suffix is specified then milliseconds is assumed.
>   
> 
> {code}
> I don't think the value of this config is used only during DataNode startup. 
> I think it's used whenever checking volumes.
> The description is misleading so we need to correct it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-13949) Correct the description of dfs.datanode.disk.check.timeout in hdfs-default.xml

2018-10-09 Thread Nanda kumar (JIRA) 


[ 
https://issues.apache.org/jira/browse/HDFS-13949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644018#comment-16644018
 ] 

Nanda kumar commented on HDFS-13949:


[~brfrn169], We use {{dfs.datanode.disk.check.timeout}} in two places; 
DatasetVolumeChecker#checkAllVolumes and StorageLocationChecker#check.

{{StorageLocationChecker#check}} is called from {{DataNode#makeInstance}} which 
is as part of datanode start-up.

{{DatasetVolumeChecker#checkAllVolumes}} is called from 
{{DataNode#checkDiskError}} when we do {{DataNode#initBlockPool}}, this happens 
when a datanode is able to successfully register with a NameNode. This should 
also be ideally considered as datanode start-up, as it is done only when we 
first register with the NameNode.

Is there anywhere else this property has been used and I'm missing it?

> Correct the description of dfs.datanode.disk.check.timeout in hdfs-default.xml
> --
>
> Key: HDFS-13949
> URL: https://issues.apache.org/jira/browse/HDFS-13949
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Minor
> Attachments: HDFS-13949.1.patch
>
>
> The description of dfs.datanode.disk.check.timeout in hdfs-default.xml is as 
> follows:
> {code}
> 
>   dfs.datanode.disk.check.timeout
>   10m
>   
> Maximum allowed time for a disk check to complete during DataNode
> startup. If the check does not complete within this time interval
> then the disk is declared as failed. This setting supports
> multiple time unit suffixes as described in dfs.heartbeat.interval.
> If no suffix is specified then milliseconds is assumed.
>   
> 
> {code}
> I don't think the value of this config is used only during DataNode startup. 
> I think it's used whenever checking volumes.
> The description is misleading so we need to correct it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-13949) Correct the description of dfs.datanode.disk.check.timeout in hdfs-default.xml

2018-10-09 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HDFS-13949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643987#comment-16643987
 ] 

Hadoop QA commented on HDFS-13949:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
39m  6s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 12s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 30s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
31s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 38s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
|   | hadoop.fs.TestHdfsNativeCodeLoader |
|   | hadoop.hdfs.TestLeaseRecovery2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | HDFS-13949 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12942063/HDFS-13949.1.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  xml  |
| uname | Linux de1092ac12fb 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 600438b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/25241/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/25241/testReport/ |
| Max. process+thread count | 3871 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/25241/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org |


This message was automatically generated.



> Correct the description of dfs.datanode.disk.check.timeout in hdfs-default.xml
>

[jira] [Commented] (HDFS-13949) Correct the description of dfs.datanode.disk.check.timeout in hdfs-default.xml

2018-10-01 Thread Toshihiro Suzuki (JIRA) 


[ 
https://issues.apache.org/jira/browse/HDFS-13949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634843#comment-16634843
 ] 

Toshihiro Suzuki commented on HDFS-13949:
-

I just attached a patch to correct the description. Could someone please review 
it?

> Correct the description of dfs.datanode.disk.check.timeout in hdfs-default.xml
> --
>
> Key: HDFS-13949
> URL: https://issues.apache.org/jira/browse/HDFS-13949
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation
>Reporter: Toshihiro Suzuki
>Priority: Minor
> Attachments: HDFS-13949.1.patch
>
>
> The description of dfs.datanode.disk.check.timeout in hdfs-default.xml is as 
> follows:
> {code}
> 
>   dfs.datanode.disk.check.timeout
>   10m
>   
> Maximum allowed time for a disk check to complete during DataNode
> startup. If the check does not complete within this time interval
> then the disk is declared as failed. This setting supports
> multiple time unit suffixes as described in dfs.heartbeat.interval.
> If no suffix is specified then milliseconds is assumed.
>   
> 
> {code}
> I don't think the value of this config is used only during DataNode startup. 
> I think it's used whenever checking volumes.
> The description is misleading so we need to correct it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org