[jira] [Commented] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299614#comment-16299614
]
John Zhuge commented on HDFS-12528:
---
Unfortunately I won't have time for a while. Unassign myself.
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
> Attachments: HDFS-12528.000.patch
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087
> not found
> {code}
> This error is handled as an "unknown response" by the BlockReaderFactory [1],
> which disables short-circuit reads for 10 minutes [2] for the client.
> These 10 minutes without SCR can have a big performance impact for the client
> operations. In this particular case ("Meta file not found") it would suffice
> to return null without disabling SCR. This particular block read would fall
> back to the normal, non-short-circuited, path and other SCR requests would
> continue to work as expected.
> It might also be interesting to be able to control how long SCR is disabled
> for in the "unknown response" case. 10 minutes seems a bit to long and not
> being able to change that is a problem.
> [1]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646
> [2]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge reassigned HDFS-12528:
-
Assignee: (was: John Zhuge)
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
> Attachments: HDFS-12528.000.patch
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087
> not found
> {code}
> This error is handled as an "unknown response" by the BlockReaderFactory [1],
> which disables short-circuit reads for 10 minutes [2] for the client.
> These 10 minutes without SCR can have a big performance impact for the client
> operations. In this particular case ("Meta file not found") it would suffice
> to return null without disabling SCR. This particular block read would fall
> back to the normal, non-short-circuited, path and other SCR requests would
> continue to work as expected.
> It might also be interesting to be able to control how long SCR is disabled
> for in the "unknown response" case. 10 minutes seems a bit to long and not
> being able to change that is a problem.
> [1]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646
> [2]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299600#comment-16299600
]
Xiao Chen commented on HDFS-12528:
--
Sorry, stupid me. I guess I didn't mvn clean when rebuilding with the HDFS-3296
patch... (Thanks for fixing that!)
Will play with the test and come back soon. BTW are you planning to work on
this issue John
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
>Assignee: John Zhuge
> Attachments: HDFS-12528.000.patch
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087
> not found
> {code}
> This error is handled as an "unknown response" by the BlockReaderFactory [1],
> which disables short-circuit reads for 10 minutes [2] for the client.
> These 10 minutes without SCR can have a big performance impact for the client
> operations. In this particular case ("Meta file not found") it would suffice
> to return null without disabling SCR. This particular block read would fall
> back to the normal, non-short-circuited, path and other SCR requests would
> continue to work as expected.
> It might also be interesting to be able to control how long SCR is disabled
> for in the "unknown response" case. 10 minutes seems a bit to long and not
> being able to change that is a problem.
> [1]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646
> [2]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299600#comment-16299600
]
Xiao Chen edited comment on HDFS-12528 at 12/21/17 5:45 AM:
Sorry, stupid me. I guess I didn't mvn clean when rebuilding with the HDFS-3296
patch... (Thanks for fixing that!)
Will play with the test and come back soon. BTW are you planning to work on
this issue John?
was (Author: xiaochen):
Sorry, stupid me. I guess I didn't mvn clean when rebuilding with the HDFS-3296
patch... (Thanks for fixing that!)
Will play with the test and come back soon. BTW are you planning to work on
this issue John
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
>Assignee: John Zhuge
> Attachments: HDFS-12528.000.patch
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087
> not found
> {code}
> This error is handled as an "unknown response" by the BlockReaderFactory [1],
> which disables short-circuit reads for 10 minutes [2] for the client.
> These 10 minutes without SCR can have a big performance impact for the client
> operations. In this particular case ("Meta file not found") it would suffice
> to return null without disabling SCR. This particular block read would fall
> back to the normal, non-short-circuited, path and other SCR requests would
> continue to work as expected.
> It might also be interesting to be able to control how long SCR is disabled
> for in the "unknown response" case. 10 minutes seems a bit to long and not
> being able to change that is a problem.
> [1]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646
> [2]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up
[ https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299596#comment-16299596 ] Jianfei Jiang edited comment on HDFS-12935 at 12/21/17 5:39 AM: Thanks [~brahmareddy] so much for your comment and adding me as a contributor. It's my honor. I have read HowToContribute and learnt a lot. After viewing the relative code. The following commands may have the some issue and I am going to handle. setSafeMode saveNamespace restoreFailedStorage refreshNodes setBalancerBandwidth finalizeUpgrade refreshServiceAcl refreshUserToGroupsMappings refreshSuperUserGroupsConfiguration refreshCallQueue was (Author: jiangjianfei): Thanks [~brahmareddy] so much for your comment and adding me as a contributor. It is my pleasure. I have read HowToContribute and learnt a lot. After viewing the relative code. The following commands may have the some issue and I am going to handle. setSafeMode saveNamespace restoreFailedStorage refreshNodes setBalancerBandwidth finalizeUpgrade refreshServiceAcl refreshUserToGroupsMappings refreshSuperUserGroupsConfiguration refreshCallQueue > Get ambiguous result for DFSAdmin command in HA mode when only one namenode > is up > - > > Key: HDFS-12935 > URL: https://issues.apache.org/jira/browse/HDFS-12935 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 3.0.0-beta1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang > Attachments: HDFS_12935.001.patch > > > In HA mode, if one namenode is down, most of functions can still work. When > considering the following two occasions: > (1)nn1 up and nn2 down > (2)nn1 down and nn2 up > These two occasions should be equivalent. However, some of the DFSAdmin > commands will have ambiguous results. The commands can be send successfully > to the up namenode and are always functionally useful only when nn1 is up > regardless of exception (IOException when connecting to the down namenode > nn2). If only nn2 is up, the commands have no use at all and only exception > to connect nn1 can be found. > See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to > set balancer bandwidth value for datanodes as an example. It works and all > the datanodes can get the setting values only when nn1 is up. If only nn2 is > up, the command throws exception directly and no datanode get the bandwidth > setting. Approximately ten DFSAdmin commands use the similar logical process > and may be ambiguous. > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345 > *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820* > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei02:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234 > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei01:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up
[ https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299596#comment-16299596 ] Jianfei Jiang commented on HDFS-12935: -- Thanks [~brahmareddy] so much for your comment and adding me as a contributor. It is my pleasure. I have read HowToContribute and learnt a lot. After viewing the relative code. The following commands may have the some issue and I am going to handle. setSafeMode saveNamespace restoreFailedStorage refreshNodes setBalancerBandwidth finalizeUpgrade refreshServiceAcl refreshUserToGroupsMappings refreshSuperUserGroupsConfiguration refreshCallQueue > Get ambiguous result for DFSAdmin command in HA mode when only one namenode > is up > - > > Key: HDFS-12935 > URL: https://issues.apache.org/jira/browse/HDFS-12935 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 3.0.0-beta1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang > Attachments: HDFS_12935.001.patch > > > In HA mode, if one namenode is down, most of functions can still work. When > considering the following two occasions: > (1)nn1 up and nn2 down > (2)nn1 down and nn2 up > These two occasions should be equivalent. However, some of the DFSAdmin > commands will have ambiguous results. The commands can be send successfully > to the up namenode and are always functionally useful only when nn1 is up > regardless of exception (IOException when connecting to the down namenode > nn2). If only nn2 is up, the commands have no use at all and only exception > to connect nn1 can be found. > See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to > set balancer bandwidth value for datanodes as an example. It works and all > the datanodes can get the setting values only when nn1 is up. If only nn2 is > up, the command throws exception directly and no datanode get the bandwidth > setting. Approximately ten DFSAdmin commands use the similar logical process > and may be ambiguous. > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345 > *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820* > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei02:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234 > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei01:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299589#comment-16299589
]
John Zhuge commented on HDFS-12528:
---
Yes, that would shorten the path greatly.
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
>Assignee: John Zhuge
> Attachments: HDFS-12528.000.patch
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087
> not found
> {code}
> This error is handled as an "unknown response" by the BlockReaderFactory [1],
> which disables short-circuit reads for 10 minutes [2] for the client.
> These 10 minutes without SCR can have a big performance impact for the client
> operations. In this particular case ("Meta file not found") it would suffice
> to return null without disabling SCR. This particular block read would fall
> back to the normal, non-short-circuited, path and other SCR requests would
> continue to work as expected.
> It might also be interesting to be able to control how long SCR is disabled
> for in the "unknown response" case. 10 minutes seems a bit to long and not
> being able to change that is a problem.
> [1]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646
> [2]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12955) [SPS]: Move SPS classes to a separate package
[ https://issues.apache.org/jira/browse/HDFS-12955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R reassigned HDFS-12955: --- Assignee: Rakesh R Description: For clean modularization, it would be good if we moved SPS related classes into its own package. (was: For clean modularization, it would be good if we moved SPS related classes into its won package) > [SPS]: Move SPS classes to a separate package > - > > Key: HDFS-12955 > URL: https://issues.apache.org/jira/browse/HDFS-12955 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nn >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Rakesh R >Priority: Trivial > > For clean modularization, it would be good if we moved SPS related classes > into its own package. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12869) Ozone: Service Discovery: RPC endpoint in KSM for getServiceList
[
https://issues.apache.org/jira/browse/HDFS-12869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299549#comment-16299549
]
Anu Engineer commented on HDFS-12869:
-
+1, pending jenkins. There is a small nit:
* nit: In function
{{KeySpaceManagerProtocolServerSideTranslatorPB#getServiceList}} the
{{ServiceException}} is never thrown, you can fix this while committing, no
need for another patch.
> Ozone: Service Discovery: RPC endpoint in KSM for getServiceList
>
>
> Key: HDFS-12869
> URL: https://issues.apache.org/jira/browse/HDFS-12869
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
> Attachments: HDFS-12869-HDFS-7240.000.patch,
> HDFS-12869-HDFS-7240.001.patch, HDFS-12869-HDFS-7240.002.patch,
> HDFS-12869-HDFS-7240.003.patch
>
>
> A new RPC call to be added to KSM which will return the list of Services that
> are there in Ozone cluster, this will be used by OzoneClient for establishing
> the connection.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299537#comment-16299537
]
Xiao Chen edited comment on HDFS-12528 at 12/21/17 4:25 AM:
{{/var/folders/6l/7hfzdv912jvclwrzyfndwjn8gp/T/socks.1513830184519.1944340260/testShortCircuitCacheUnbuffer._PORT}}
Is the intention of HDFS-3296 patch to make this {{/tmp/xxx}} ?
was (Author: xiaochen):
{{/var/folders/6l/7hfzdv912jvclwrzyfndwjn8gp/T/socks.1513830184519.1944340260/testShortCircuitCacheUnbuffer._PORT}}
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
>Assignee: John Zhuge
> Attachments: HDFS-12528.000.patch
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087
> not found
> {code}
> This error is handled as an "unknown response" by the BlockReaderFactory [1],
> which disables short-circuit reads for 10 minutes [2] for the client.
> These 10 minutes without SCR can have a big performance impact for the client
> operations. In this particular case ("Meta file not found") it would suffice
> to return null without disabling SCR. This particular block read would fall
> back to the normal, non-short-circuited, path and other SCR requests would
> continue to work as expected.
> It might also be interesting to be able to control how long SCR is disabled
> for in the "unknown response" case. 10 minutes seems a bit to long and not
> being able to change that is a problem.
> [1]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646
> [2]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299537#comment-16299537
]
Xiao Chen commented on HDFS-12528:
--
{{/var/folders/6l/7hfzdv912jvclwrzyfndwjn8gp/T/socks.1513830184519.1944340260/testShortCircuitCacheUnbuffer._PORT}}
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
>Assignee: John Zhuge
> Attachments: HDFS-12528.000.patch
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087
> not found
> {code}
> This error is handled as an "unknown response" by the BlockReaderFactory [1],
> which disables short-circuit reads for 10 minutes [2] for the client.
> These 10 minutes without SCR can have a big performance impact for the client
> operations. In this particular case ("Meta file not found") it would suffice
> to return null without disabling SCR. This particular block read would fall
> back to the normal, non-short-circuited, path and other SCR requests would
> continue to work as expected.
> It might also be interesting to be able to control how long SCR is disabled
> for in the "unknown response" case. 10 minutes seems a bit to long and not
> being able to change that is a problem.
> [1]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646
> [2]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299531#comment-16299531
]
John Zhuge commented on HDFS-12528:
---
[~xiaochen] Puzzled. The hadoop-hdfs/pom.xml changes in HDFS-3296.004.patch
should reduce the length of domain socket path string. What is the path string?
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
>Assignee: John Zhuge
> Attachments: HDFS-12528.000.patch
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087
> not found
> {code}
> This error is handled as an "unknown response" by the BlockReaderFactory [1],
> which disables short-circuit reads for 10 minutes [2] for the client.
> These 10 minutes without SCR can have a big performance impact for the client
> operations. In this particular case ("Meta file not found") it would suffice
> to return null without disabling SCR. This particular block read would fall
> back to the normal, non-short-circuited, path and other SCR requests would
> continue to work as expected.
> It might also be interesting to be able to control how long SCR is disabled
> for in the "unknown response" case. 10 minutes seems a bit to long and not
> being able to change that is a problem.
> [1]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646
> [2]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12860) StripedBlockUtil#getRangesInternalBlocks throws exception for the block group size larger than 2GB
[
https://issues.apache.org/jira/browse/HDFS-12860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299516#comment-16299516
]
genericqa commented on HDFS-12860:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
50s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-hdfs-project: The patch generated 3 new +
24 unchanged - 3 fixed = 27 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
24s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}116m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
36s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}190m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
| | hadoop.cli.TestErasureCodingCLI |
| | hadoop.hdfs.server.blockmanagement.TestSequentialBlockGroupId |
| | hadoop.hdfs.server.namenode.TestUpgradeDomainBlockPlacementPolicy |
| | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.server.namenode.TestFSDirectory |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.s
[jira] [Commented] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299515#comment-16299515
]
Xiao Chen commented on HDFS-12528:
--
Thanks all for the discussions, and [~jzhuge] for a test patch.
Hi John,
I tried to run {{TestBlockReaderFactory}} with your patch on my mac, with
freshly built native libs. Got:
{noformat}
java.net.SocketException: error computing UNIX domain socket path: path too
long. The longest UNIX domain socket path possible on this host is 103 bytes.
at org.apache.hadoop.net.unix.DomainSocket.bind0(Native Method)
at
org.apache.hadoop.net.unix.DomainSocket.bindAndListen(DomainSocket.java:211)
at
org.apache.hadoop.hdfs.net.DomainPeerServer.(DomainPeerServer.java:40)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:1189)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:1156)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1401)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:497)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2778)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2681)
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1643)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:885)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:497)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:456)
at
org.apache.hadoop.hdfs.client.impl.TestBlockReaderFactory.testShortCircuitCacheTemporaryFailure(TestBlockReaderFactory.java:241)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{noformat}
Searched around and found HDFS-3296, but even with the patches there I still
run into this error. Any ideas how to get over it?
Regarding the behavior, not sure who's the SCR expert we should ping. I'm
inclined to agree with [~cheersyang] that we can make the disable behavior
configurable, and not changing the default behavior. (Don't think there is a
good way to differentiate between the Meta file not found IOE and other IOEs.)
We can also make the 10 minute disable interval configurable as Andre suggested
on the initial report.
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
>Assignee: John Zhuge
> Attachments: HDFS-12528.000.patch
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-149
[jira] [Updated] (HDFS-12869) Ozone: Service Discovery: RPC endpoint in KSM for getServiceList
[ https://issues.apache.org/jira/browse/HDFS-12869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nanda kumar updated HDFS-12869: --- Attachment: HDFS-12869-HDFS-7240.003.patch > Ozone: Service Discovery: RPC endpoint in KSM for getServiceList > > > Key: HDFS-12869 > URL: https://issues.apache.org/jira/browse/HDFS-12869 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nanda kumar >Assignee: Nanda kumar > Attachments: HDFS-12869-HDFS-7240.000.patch, > HDFS-12869-HDFS-7240.001.patch, HDFS-12869-HDFS-7240.002.patch, > HDFS-12869-HDFS-7240.003.patch > > > A new RPC call to be added to KSM which will return the list of Services that > are there in Ozone cluster, this will be used by OzoneClient for establishing > the connection. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12869) Ozone: Service Discovery: RPC endpoint in KSM for getServiceList
[ https://issues.apache.org/jira/browse/HDFS-12869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299510#comment-16299510 ] Nanda kumar commented on HDFS-12869: Thanks for the update [~anu], I have rebased the patch. > Ozone: Service Discovery: RPC endpoint in KSM for getServiceList > > > Key: HDFS-12869 > URL: https://issues.apache.org/jira/browse/HDFS-12869 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nanda kumar >Assignee: Nanda kumar > Attachments: HDFS-12869-HDFS-7240.000.patch, > HDFS-12869-HDFS-7240.001.patch, HDFS-12869-HDFS-7240.002.patch, > HDFS-12869-HDFS-7240.003.patch > > > A new RPC call to be added to KSM which will return the list of Services that > are there in Ozone cluster, this will be used by OzoneClient for establishing > the connection. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11847) Enhance dfsadmin listOpenFiles command to list files blocking datanode decommissioning
[
https://issues.apache.org/jira/browse/HDFS-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299505#comment-16299505
]
genericqa commented on HDFS-11847:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
50s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
810 unchanged - 1 fixed = 811 total (was 811) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 17s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
26s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}123m 59s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.protocol.TestAnnotations |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.0
[jira] [Commented] (HDFS-12812) Ozone: dozone: initialize scm directory on cluster startup
[
https://issues.apache.org/jira/browse/HDFS-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299506#comment-16299506
]
Nanda kumar commented on HDFS-12812:
HDFS-12741 brings in support for {{ksm --createObjectStore}}, which got
committed recently. With SCM initialization dozone should also include ksm
createObjectStore command.
> Ozone: dozone: initialize scm directory on cluster startup
> --
>
> Key: HDFS-12812
> URL: https://issues.apache.org/jira/browse/HDFS-12812
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Attachments: HADOOP-12812-HDFS-7240.001.patch,
> HDFS-12812-HDFS-7240.002.patch
>
>
> HDFS-12739 fixed the scm, but after the patch it couldn't be started without
> a separated `hdfs scm -init` any more. Unfortunatelly it breaks the
> docker-compose functionality.
> This is handled int the starter script of the base runner docker image for
> namenode. I also fixed this in the runner docker image
> (https://github.com/elek/hadoop/commit/b347eb4bfca37d84dbcdd8c4bf353219d876a9b7)
> will upload the improved patch to the HADOOP-14898.
> In this patch I just add a new environment variable to init the scm if the
> version file doesn't exist.
> To test:
> Do a full build and go to the dev-support/compose/ozone:
> ```
> docker-compose pull
> docker-compose up -d
> ```
> Note: the pull is important as the new docker image -- which could handle the
> new environment variable -- should be used
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12934) RBF: Federation supports global quota
[
https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299467#comment-16299467
]
Íñigo Goiri commented on HDFS-12934:
BTW, there is also the {{NamenodeHeartbeatService}} which queries the NN and
puts the updated information into the State Store.
I don't have a preference for reusing this or creating this new service to
update each local cache; but I think it's worth giving it a thought.
> RBF: Federation supports global quota
> -
>
> Key: HDFS-12934
> URL: https://issues.apache.org/jira/browse/HDFS-12934
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Labels: RBF
> Attachments: RBF support global quota.pdf
>
>
> Now federation doesn't support set the global quota for each folder.
> Currently the quota will be applied for each subcluster under the specified
> folder via RPC call.
> It will be very useful for users that federation can support setting global
> quota and exposing the command of this.
> In a federated environment, a folder can be spread across multiple
> subclusters. For this reason, we plan to solve this by following way:
> # Set global quota across each subcluster. We don't allow each subcluster can
> exceed maximun quota value.
> # We need to construct one cache map for storing the sum
> quota usage of these subclusters under federation folder. Every time we want
> to do WRITE operation under specified folder, we will get its quota usage
> from cache and verify its quota. If quota exceeded, throw exception,
> otherwise update its quota usage in cache when finishing operations.
> The quota will be set to mount table and as a new field in mount table. The
> set/unset command will be like:
> {noformat}
> hdfs dfsrouteradmin -setQuota -ns -ss
> hdfs dfsrouteradmin -clrQuota
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[
https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299451#comment-16299451
]
genericqa commented on HDFS-12942:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
42s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}130m 26s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}178m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestErasureCodingMultipleRacks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12942 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903127/HDFS-12942.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 62564cba1e5d 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 382215c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/22478/artifact/out/branch-findb
[jira] [Commented] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[
https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299434#comment-16299434
]
genericqa commented on HDFS-12942:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
6s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}125m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}177m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12942 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903126/HDFS-12942.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b4efef4fbfd2 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 382215c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/22477/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22477/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/22477/testReport/ |
| Max. process+thread count | 3095 (vs. ulimit of 5000) |
| modules | C: hadoop-hdfs
[jira] [Commented] (HDFS-12934) RBF: Federation supports global quota
[
https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299431#comment-16299431
]
Yiqun Lin commented on HDFS-12934:
--
Thanks for taking the look of the design doc, [~elgoiri].
bq. I think we should have flags to enable the quota system at the Router level
Good idea. Agree on this. And quota system should be disabled by default.
{quote}
I guess we also want to implement the setQuotaSpace().
{quote}
The API {{ClientProtocol#setQuota(String path, long namespaceQuota, long
storagespaceQuota, StorageType type)}} can set space quota. If we don't want to
set namespaceQuota, we can pass an unset flag like
{{HdfsConstants.QUOTA_DONT_SET}}.
bq. I didn't get what you meant about ClientProtocol#getQuota(); will we
implement it or not?
I am thinking of this again. This should be implemented. We can use this API to
get quota usage of given path. In my initial design doc, I plan to create new
RouterRPCClient for making the RPC call. But actually we can implement this in
RouterRpcServer and reuse this API.
i will attach a new doc for updating the content you pointed out. Now I think
we almostly reach an agreement for quota supporting in RBF. I will start for
working on this right now. Thanks for discussions, everyone, :).
> RBF: Federation supports global quota
> -
>
> Key: HDFS-12934
> URL: https://issues.apache.org/jira/browse/HDFS-12934
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Labels: RBF
> Attachments: RBF support global quota.pdf
>
>
> Now federation doesn't support set the global quota for each folder.
> Currently the quota will be applied for each subcluster under the specified
> folder via RPC call.
> It will be very useful for users that federation can support setting global
> quota and exposing the command of this.
> In a federated environment, a folder can be spread across multiple
> subclusters. For this reason, we plan to solve this by following way:
> # Set global quota across each subcluster. We don't allow each subcluster can
> exceed maximun quota value.
> # We need to construct one cache map for storing the sum
> quota usage of these subclusters under federation folder. Every time we want
> to do WRITE operation under specified folder, we will get its quota usage
> from cache and verify its quota. If quota exceeded, throw exception,
> otherwise update its quota usage in cache when finishing operations.
> The quota will be set to mount table and as a new field in mount table. The
> set/unset command will be like:
> {noformat}
> hdfs dfsrouteradmin -setQuota -ns -ss
> hdfs dfsrouteradmin -clrQuota
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12943) Consistent Reads from Standby Node
[ https://issues.apache.org/jira/browse/HDFS-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299398#comment-16299398 ] Virajith Jalaparti commented on HDFS-12943: --- Hi [~shv], thanks for posting the design document. One thing that wasn't clear to me from the design doc itself was what's the function of the Observer Nodes. Are these what the clients actually use to read, instead of the real SBN? Further, what's the goal to of having them? Is it to reduce the load on the SBN further or graceful degradation during failures of NN/SBN? > Consistent Reads from Standby Node > -- > > Key: HDFS-12943 > URL: https://issues.apache.org/jira/browse/HDFS-12943 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs >Reporter: Konstantin Shvachko > Attachments: ConsistentReadsFromStandbyNode.pdf > > > StandbyNode in HDFS is a replica of the active NameNode. The states of the > NameNodes are coordinated via the journal. It is natural to consider > StandbyNode as a read-only replica. As with any replicated distributed system > the problem of stale reads should be resolved. Our main goal is to provide > reads from standby in a consistent way in order to enable a wide range of > existing applications running on top of HDFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12860) StripedBlockUtil#getRangesInternalBlocks throws exception for the block group size larger than 2GB
[
https://issues.apache.org/jira/browse/HDFS-12860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-12860:
-
Summary: StripedBlockUtil#getRangesInternalBlocks throws exception for the
block group size larger than 2GB (was: TeraSort failed on erasure coding
directory)
> StripedBlockUtil#getRangesInternalBlocks throws exception for the block group
> size larger than 2GB
> --
>
> Key: HDFS-12860
> URL: https://issues.apache.org/jira/browse/HDFS-12860
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-12860.00.patch, HDFS-12860.00.patch
>
>
> Running terasort on a cluster with 8 datanodes, 256GB data, using
> RS-3-2-1024k.
> The test data was generated by {{teragen}} with 32 mappers.
> The terasort benchmark fails with the following stack trace:
> {code}
> 17/11/27 14:44:31 INFO mapreduce.Job: map 45% reduce 0%
> 17/11/27 14:44:33 INFO mapreduce.Job: Task Id :
> attempt_1510080297865_0160_m_08_0, Status : FAILED
> Error: java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil$VerticalRange.(StripedBlockUtil.java:701)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.getRangesForInternalBlocks(StripedBlockUtil.java:442)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.divideOneStripe(StripedBlockUtil.java:311)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:308)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:391)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:813)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.examples.terasort.TeraInputFormat$TeraRecordReader.nextKeyValue(TeraInputFormat.java:257)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:562)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12955) [SPS]: Move SPS classes to a separate package
Uma Maheswara Rao G created HDFS-12955: -- Summary: [SPS]: Move SPS classes to a separate package Key: HDFS-12955 URL: https://issues.apache.org/jira/browse/HDFS-12955 Project: Hadoop HDFS Issue Type: Sub-task Components: nn Affects Versions: HDFS-10285 Reporter: Uma Maheswara Rao G Priority: Trivial For clean modularization, it would be good if we moved SPS related classes into its won package -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12949) Fix findbugs warning in ImageWriter.java
[
https://issues.apache.org/jira/browse/HDFS-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299359#comment-16299359
]
Hudson commented on HDFS-12949:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13413 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13413/])
HDFS-12949. Fix findbugs warning in ImageWriter.java. (aajisaka: rev
5ab632baf52f0ecc737845051b382f68bf1385bb)
* (edit) hadoop-tools/hadoop-fs2img/pom.xml
> Fix findbugs warning in ImageWriter.java
>
>
> Key: HDFS-12949
> URL: https://issues.apache.org/jira/browse/HDFS-12949
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
> Fix For: 3.1.0
>
> Attachments: HDFS-12949.001.patch
>
>
> There are 1 findbugs warning in hadoop-fs2img module.
> {noformat}
> OBL new
> org.apache.hadoop.hdfs.server.namenode.ImageWriter(ImageWriter$Options) may
> fail to clean up java.io.OutputStream on checked exception
> Bug type OBL_UNSATISFIED_OBLIGATION_EXCEPTION_EDGE (click for details)
> In class org.apache.hadoop.hdfs.server.namenode.ImageWriter
> In method new
> org.apache.hadoop.hdfs.server.namenode.ImageWriter(ImageWriter$Options)
> Reference type java.io.OutputStream
> 1 instances of obligation remaining
> Obligation to clean up resource created at ImageWriter.java:[line 184] is not
> discharged
> Path continues at ImageWriter.java:[line 185]
> Path continues at ImageWriter.java:[line 186]
> Path continues at ImageWriter.java:[line 187]
> Path continues at ImageWriter.java:[line 188]
> Remaining obligations: {OutputStream x 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12863) Ozone: Avoid configuration.get() call for each container report
[
https://issues.apache.org/jira/browse/HDFS-12863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299358#comment-16299358
]
Akira Ajisaka commented on HDFS-12863:
--
Thank you, [~anu]!
> Ozone: Avoid configuration.get() call for each container report
> ---
>
> Key: HDFS-12863
> URL: https://issues.apache.org/jira/browse/HDFS-12863
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Minor
> Fix For: HDFS-7240
>
> Attachments: HDFS-12863-HDFS-7240.001.patch,
> HDFS-12863-HDFS-7240.002.patch
>
>
> In ContainerReportManagerImpl.java,
> {{OzoneClientUtils.getScmHeartbeatInterval(config)}} is called for each
> container report. We should cache the interval and avoid calling
> {{Configuration.get(name)}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12949) Fix findbugs warning in ImageWriter.java
[
https://issues.apache.org/jira/browse/HDFS-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-12949:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
Committed this to trunk. Thanks [~chris.douglas] for the review.
> Fix findbugs warning in ImageWriter.java
>
>
> Key: HDFS-12949
> URL: https://issues.apache.org/jira/browse/HDFS-12949
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
> Fix For: 3.1.0
>
> Attachments: HDFS-12949.001.patch
>
>
> There are 1 findbugs warning in hadoop-fs2img module.
> {noformat}
> OBL new
> org.apache.hadoop.hdfs.server.namenode.ImageWriter(ImageWriter$Options) may
> fail to clean up java.io.OutputStream on checked exception
> Bug type OBL_UNSATISFIED_OBLIGATION_EXCEPTION_EDGE (click for details)
> In class org.apache.hadoop.hdfs.server.namenode.ImageWriter
> In method new
> org.apache.hadoop.hdfs.server.namenode.ImageWriter(ImageWriter$Options)
> Reference type java.io.OutputStream
> 1 instances of obligation remaining
> Obligation to clean up resource created at ImageWriter.java:[line 184] is not
> discharged
> Path continues at ImageWriter.java:[line 185]
> Path continues at ImageWriter.java:[line 186]
> Path continues at ImageWriter.java:[line 187]
> Path continues at ImageWriter.java:[line 188]
> Remaining obligations: {OutputStream x 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12953) XORRawDecoder.doDecode throws NullPointerException
[
https://issues.apache.org/jira/browse/HDFS-12953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu reassigned HDFS-12953:
Assignee: Lei (Eddy) Xu
> XORRawDecoder.doDecode throws NullPointerException
> --
>
> Key: HDFS-12953
> URL: https://issues.apache.org/jira/browse/HDFS-12953
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>
> Thanks [~danielpol] report on HDFS-12860.
> {noformat}
> 17/11/30 04:19:55 INFO mapreduce.Job: map 0% reduce 0%
> 17/11/30 04:20:01 INFO mapreduce.Job: Task Id :
> attempt_1512036058655_0003_m_02_0, Status : FAILED
> Error: java.lang.NullPointerException
> at
> org.apache.hadoop.io.erasurecode.rawcoder.XORRawDecoder.doDecode(XORRawDecoder.java:83)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:106)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
> at
> org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:423)
> at
> org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
> at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:382)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:318)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:391)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:813)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.examples.terasort.TeraInputFormat$TeraRecordReader.nextKeyValue(TeraInputFormat.java:257)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:563)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:794)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12954) Ozone: Container : Add key versioning support-3
Chen Liang created HDFS-12954:
-
Summary: Ozone: Container : Add key versioning support-3
Key: HDFS-12954
URL: https://issues.apache.org/jira/browse/HDFS-12954
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Affects Versions: HDFS-7240
Reporter: Chen Liang
Assignee: Chen Liang
A new version of a key is effectively overwriting some consecutive range of
bytes in the entire key offset range. For each version, we need to keep exactly
what the range is in order for the IO vector to work.
Currently, since we only write from the start (offset = 0), so offset range of
a version is only up to the key data size field when the version gets
committed. But currently we only keep one single key data size variable.(see
{{KeyManagerImpl#commitKey}}). We need to know the corresponding key data size
for each version. This JIRA is to the tracking of offset range for each version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12953) XORRawDecoder.doDecode throws NullPointerException
Lei (Eddy) Xu created HDFS-12953:
Summary: XORRawDecoder.doDecode throws NullPointerException
Key: HDFS-12953
URL: https://issues.apache.org/jira/browse/HDFS-12953
Project: Hadoop HDFS
Issue Type: Bug
Components: erasure-coding
Affects Versions: 3.0.0
Reporter: Lei (Eddy) Xu
Thanks [~danielpol] report on HDFS-12860.
{noformat}
17/11/30 04:19:55 INFO mapreduce.Job: map 0% reduce 0%
17/11/30 04:20:01 INFO mapreduce.Job: Task Id :
attempt_1512036058655_0003_m_02_0, Status : FAILED
Error: java.lang.NullPointerException
at
org.apache.hadoop.io.erasurecode.rawcoder.XORRawDecoder.doDecode(XORRawDecoder.java:83)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:106)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:423)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:382)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:318)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:391)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:813)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.examples.terasort.TeraInputFormat$TeraRecordReader.nextKeyValue(TeraInputFormat.java:257)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:563)
at
org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:794)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11847) Enhance dfsadmin listOpenFiles command to list files blocking datanode decommissioning
[
https://issues.apache.org/jira/browse/HDFS-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11847:
--
Attachment: HDFS-11847.03.patch
> Enhance dfsadmin listOpenFiles command to list files blocking datanode
> decommissioning
> --
>
> Key: HDFS-11847
> URL: https://issues.apache.org/jira/browse/HDFS-11847
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11847.01.patch, HDFS-11847.02.patch,
> HDFS-11847.03.patch
>
>
> HDFS-10480 adds {{listOpenFiles}} option is to {{dfsadmin}} command to list
> all the open files in the system.
> Additionally, it would be very useful to only list open files that are
> blocking the DataNode decommissioning. With thousand+ node clusters, where
> there might be machines added and removed regularly for maintenance, any
> option to monitor and debug decommissioning status is very helpful. Proposal
> here is to add suboptions to {{listOpenFiles}} for the above case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12860) TeraSort failed on erasure coding directory
[
https://issues.apache.org/jira/browse/HDFS-12860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-12860:
-
Status: Patch Available (was: Open)
> TeraSort failed on erasure coding directory
> ---
>
> Key: HDFS-12860
> URL: https://issues.apache.org/jira/browse/HDFS-12860
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-12860.00.patch, HDFS-12860.00.patch
>
>
> Running terasort on a cluster with 8 datanodes, 256GB data, using
> RS-3-2-1024k.
> The test data was generated by {{teragen}} with 32 mappers.
> The terasort benchmark fails with the following stack trace:
> {code}
> 17/11/27 14:44:31 INFO mapreduce.Job: map 45% reduce 0%
> 17/11/27 14:44:33 INFO mapreduce.Job: Task Id :
> attempt_1510080297865_0160_m_08_0, Status : FAILED
> Error: java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil$VerticalRange.(StripedBlockUtil.java:701)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.getRangesForInternalBlocks(StripedBlockUtil.java:442)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.divideOneStripe(StripedBlockUtil.java:311)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:308)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:391)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:813)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.examples.terasort.TeraInputFormat$TeraRecordReader.nextKeyValue(TeraInputFormat.java:257)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:562)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12860) TeraSort failed on erasure coding directory
[
https://issues.apache.org/jira/browse/HDFS-12860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-12860:
-
Attachment: HDFS-12860.00.patch
> TeraSort failed on erasure coding directory
> ---
>
> Key: HDFS-12860
> URL: https://issues.apache.org/jira/browse/HDFS-12860
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-12860.00.patch, HDFS-12860.00.patch
>
>
> Running terasort on a cluster with 8 datanodes, 256GB data, using
> RS-3-2-1024k.
> The test data was generated by {{teragen}} with 32 mappers.
> The terasort benchmark fails with the following stack trace:
> {code}
> 17/11/27 14:44:31 INFO mapreduce.Job: map 45% reduce 0%
> 17/11/27 14:44:33 INFO mapreduce.Job: Task Id :
> attempt_1510080297865_0160_m_08_0, Status : FAILED
> Error: java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil$VerticalRange.(StripedBlockUtil.java:701)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.getRangesForInternalBlocks(StripedBlockUtil.java:442)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.divideOneStripe(StripedBlockUtil.java:311)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:308)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:391)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:813)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.examples.terasort.TeraInputFormat$TeraRecordReader.nextKeyValue(TeraInputFormat.java:257)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:562)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12860) TeraSort failed on erasure coding directory
[
https://issues.apache.org/jira/browse/HDFS-12860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-12860:
-
Attachment: HDFS-12860.00.patch
> TeraSort failed on erasure coding directory
> ---
>
> Key: HDFS-12860
> URL: https://issues.apache.org/jira/browse/HDFS-12860
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-12860.00.patch
>
>
> Running terasort on a cluster with 8 datanodes, 256GB data, using
> RS-3-2-1024k.
> The test data was generated by {{teragen}} with 32 mappers.
> The terasort benchmark fails with the following stack trace:
> {code}
> 17/11/27 14:44:31 INFO mapreduce.Job: map 45% reduce 0%
> 17/11/27 14:44:33 INFO mapreduce.Job: Task Id :
> attempt_1510080297865_0160_m_08_0, Status : FAILED
> Error: java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil$VerticalRange.(StripedBlockUtil.java:701)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.getRangesForInternalBlocks(StripedBlockUtil.java:442)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.divideOneStripe(StripedBlockUtil.java:311)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:308)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:391)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:813)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.examples.terasort.TeraInputFormat$TeraRecordReader.nextKeyValue(TeraInputFormat.java:257)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:562)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12860) TeraSort failed on erasure coding directory
[
https://issues.apache.org/jira/browse/HDFS-12860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299333#comment-16299333
]
Lei (Eddy) Xu commented on HDFS-12860:
--
[~danielpol] Thanks for the report! The error you reported seems to be another
bug. I will file a new JIRA after this comment.
I found the reason behind this JIRA is due to the block size is larger than 2GB
(set incorrectly during teragen), so that the Preconditions check overflow when
{{idxInInternalBlk > 2048}}:
{code}
// idxInInternalBlk is an interger, cellSize = 1MB.
Preconditions.checkState(cell.idxInInternalBlk * cellSize + cell.offset >= 0)
{code}
Upload the patch and add more debug information to related classes.
> TeraSort failed on erasure coding directory
> ---
>
> Key: HDFS-12860
> URL: https://issues.apache.org/jira/browse/HDFS-12860
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>
> Running terasort on a cluster with 8 datanodes, 256GB data, using
> RS-3-2-1024k.
> The test data was generated by {{teragen}} with 32 mappers.
> The terasort benchmark fails with the following stack trace:
> {code}
> 17/11/27 14:44:31 INFO mapreduce.Job: map 45% reduce 0%
> 17/11/27 14:44:33 INFO mapreduce.Job: Task Id :
> attempt_1510080297865_0160_m_08_0, Status : FAILED
> Error: java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil$VerticalRange.(StripedBlockUtil.java:701)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.getRangesForInternalBlocks(StripedBlockUtil.java:442)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.divideOneStripe(StripedBlockUtil.java:311)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:308)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:391)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:813)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.examples.terasort.TeraInputFormat$TeraRecordReader.nextKeyValue(TeraInputFormat.java:257)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:562)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11847) Enhance dfsadmin listOpenFiles command to list files blocking datanode decommissioning
[
https://issues.apache.org/jira/browse/HDFS-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11847:
--
Attachment: (was: HDFS-11847.03.patch)
> Enhance dfsadmin listOpenFiles command to list files blocking datanode
> decommissioning
> --
>
> Key: HDFS-11847
> URL: https://issues.apache.org/jira/browse/HDFS-11847
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11847.01.patch, HDFS-11847.02.patch
>
>
> HDFS-10480 adds {{listOpenFiles}} option is to {{dfsadmin}} command to list
> all the open files in the system.
> Additionally, it would be very useful to only list open files that are
> blocking the DataNode decommissioning. With thousand+ node clusters, where
> there might be machines added and removed regularly for maintenance, any
> option to monitor and debug decommissioning status is very helpful. Proposal
> here is to add suboptions to {{listOpenFiles}} for the above case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11847) Enhance dfsadmin listOpenFiles command to list files blocking datanode decommissioning
[
https://issues.apache.org/jira/browse/HDFS-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11847:
--
Attachment: HDFS-11847.03.patch
Thanks for the detailed review [~xiaochen]. Attached v03 patch to address the
following. Please take a look.
1. Deprecated the old API and added a new one which accepts the additional type
argument to filter the result by.
2. Updated {{FSN#listOpenFiles}} to check for {{ALL_OPEN_FILES}} type first and
then for combination filtering option later. But, the result set and the
reporting doesn't differentiate the entries by type. For this, we need to add
the type to the {{OpenFileEntry}}. Will do this.
3. About printing DataNodes details in the results, planning to take this
enhancement along with the pending item in (2) in a separate jira if you are
ok. I need to change the proto, the handling of the
4. Yes, better to return as much result as possible. Made
{{DatanodeAdminManager#processBlocksInternal}} to log the warning message on
unexpected open files and continue to the next one.
5. In {{DatanodeAdminManager#processBlocksInternal}}, the computation is at the
DataNode level. There can be multiple blocks across DNs for the same file and
the full count need to be tracked for JMX reporting purposes. So, retaining the
existing lowRedundancyBlocksInOpenFiles field. When I removed this field and
piggy backed on the {{lowRedundancyOpenFiles.size()}}, the actual count was
lesser than the expected for few tests.
6. In {{LeavingServiceStatus}} both members are needed due to (5)
7. Updated the comment for the class {{LeavingServiceStatus}}
8. {{FSN#getFilesBlockingDecom}} added hasReadLock()
9. {{TestDecommission#verifyOpenFilesBlockingDecommission}} PrintStream is now
copied before the exchange and restored to the old one. Good find.
10. {{DFS_NAMENODE_REDUNDANCY_INTERVAL_SECONDS_KEY}} value is actually in
seconds. So, it is 1000 seconds and not 1 sec. Anyways, updated this to Max
value.
> Enhance dfsadmin listOpenFiles command to list files blocking datanode
> decommissioning
> --
>
> Key: HDFS-11847
> URL: https://issues.apache.org/jira/browse/HDFS-11847
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11847.01.patch, HDFS-11847.02.patch,
> HDFS-11847.03.patch
>
>
> HDFS-10480 adds {{listOpenFiles}} option is to {{dfsadmin}} command to list
> all the open files in the system.
> Additionally, it would be very useful to only list open files that are
> blocking the DataNode decommissioning. With thousand+ node clusters, where
> there might be machines added and removed regularly for maintenance, any
> option to monitor and debug decommissioning status is very helpful. Proposal
> here is to add suboptions to {{listOpenFiles}} for the above case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12944) Update NOTICE for AssertJ dependency
[ https://issues.apache.org/jira/browse/HDFS-12944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299296#comment-16299296 ] Chris Douglas commented on HDFS-12944: -- Thanks for the clarification and reference. I agree with your read of the licensing docs. > Update NOTICE for AssertJ dependency > > > Key: HDFS-12944 > URL: https://issues.apache.org/jira/browse/HDFS-12944 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Chris Douglas >Priority: Trivial > Attachments: HDFS-12944.00.patch > > > HDFS-12665 added a dependency on the ALv2 > [AssertJ|https://github.com/joel-costigliola/assertj-core] library. We should > update the notice. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12944) Update NOTICE for AssertJ dependency
[ https://issues.apache.org/jira/browse/HDFS-12944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HDFS-12944: - Resolution: Not A Problem Status: Resolved (was: Patch Available) > Update NOTICE for AssertJ dependency > > > Key: HDFS-12944 > URL: https://issues.apache.org/jira/browse/HDFS-12944 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Chris Douglas >Priority: Trivial > Attachments: HDFS-12944.00.patch > > > HDFS-12665 added a dependency on the ALv2 > [AssertJ|https://github.com/joel-costigliola/assertj-core] library. We should > update the notice. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12949) Fix findbugs warning in ImageWriter.java
[
https://issues.apache.org/jira/browse/HDFS-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299279#comment-16299279
]
Chris Douglas commented on HDFS-12949:
--
+1 Thanks, [~ajisakaa]
> Fix findbugs warning in ImageWriter.java
>
>
> Key: HDFS-12949
> URL: https://issues.apache.org/jira/browse/HDFS-12949
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
> Attachments: HDFS-12949.001.patch
>
>
> There are 1 findbugs warning in hadoop-fs2img module.
> {noformat}
> OBL new
> org.apache.hadoop.hdfs.server.namenode.ImageWriter(ImageWriter$Options) may
> fail to clean up java.io.OutputStream on checked exception
> Bug type OBL_UNSATISFIED_OBLIGATION_EXCEPTION_EDGE (click for details)
> In class org.apache.hadoop.hdfs.server.namenode.ImageWriter
> In method new
> org.apache.hadoop.hdfs.server.namenode.ImageWriter(ImageWriter$Options)
> Reference type java.io.OutputStream
> 1 instances of obligation remaining
> Obligation to clean up resource created at ImageWriter.java:[line 184] is not
> discharged
> Path continues at ImageWriter.java:[line 185]
> Path continues at ImageWriter.java:[line 186]
> Path continues at ImageWriter.java:[line 187]
> Path continues at ImageWriter.java:[line 188]
> Remaining obligations: {OutputStream x 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-12942: -- Attachment: HDFS-12942.003.patch > Synchronization issue in FSDataSetImpl#moveBlock > > > Key: HDFS-12942 > URL: https://issues.apache.org/jira/browse/HDFS-12942 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch, > HDFS-12942.003.patch > > > FSDataSetImpl#moveBlock works in following following 3 steps: > # first creates a new replicaInfo object > # calls finalizeReplica to finalize it. > # Calls removeOldReplica to remove oldReplica. > A client can potentially append to the old replica between step 1 and 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12869) Ozone: Service Discovery: RPC endpoint in KSM for getServiceList
[ https://issues.apache.org/jira/browse/HDFS-12869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299268#comment-16299268 ] Anu Engineer commented on HDFS-12869: - [~nandakumar131] This patch is failing, Could you please rebase to the top of the tree? Thanks > Ozone: Service Discovery: RPC endpoint in KSM for getServiceList > > > Key: HDFS-12869 > URL: https://issues.apache.org/jira/browse/HDFS-12869 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nanda kumar >Assignee: Nanda kumar > Attachments: HDFS-12869-HDFS-7240.000.patch, > HDFS-12869-HDFS-7240.001.patch, HDFS-12869-HDFS-7240.002.patch > > > A new RPC call to be added to KSM which will return the list of Services that > are there in Ozone cluster, this will be used by OzoneClient for establishing > the connection. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-12942: -- Attachment: (was: HDFS-12942.003.patch) > Synchronization issue in FSDataSetImpl#moveBlock > > > Key: HDFS-12942 > URL: https://issues.apache.org/jira/browse/HDFS-12942 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch > > > FSDataSetImpl#moveBlock works in following following 3 steps: > # first creates a new replicaInfo object > # calls finalizeReplica to finalize it. > # Calls removeOldReplica to remove oldReplica. > A client can potentially append to the old replica between step 1 and 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-12942: -- Attachment: HDFS-12942.003.patch > Synchronization issue in FSDataSetImpl#moveBlock > > > Key: HDFS-12942 > URL: https://issues.apache.org/jira/browse/HDFS-12942 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch, > HDFS-12942.003.patch > > > FSDataSetImpl#moveBlock works in following following 3 steps: > # first creates a new replicaInfo object > # calls finalizeReplica to finalize it. > # Calls removeOldReplica to remove oldReplica. > A client can potentially append to the old replica between step 1 and 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-12942: -- Attachment: (was: HDFS-12942.003.patch) > Synchronization issue in FSDataSetImpl#moveBlock > > > Key: HDFS-12942 > URL: https://issues.apache.org/jira/browse/HDFS-12942 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch, > HDFS-12942.003.patch > > > FSDataSetImpl#moveBlock works in following following 3 steps: > # first creates a new replicaInfo object > # calls finalizeReplica to finalize it. > # Calls removeOldReplica to remove oldReplica. > A client can potentially append to the old replica between step 1 and 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299262#comment-16299262 ] Ajay Kumar commented on HDFS-12942: --- [~virajith], Addressed it in patch v3. Also fixed checkstyle issue from jenkins precommit. > Synchronization issue in FSDataSetImpl#moveBlock > > > Key: HDFS-12942 > URL: https://issues.apache.org/jira/browse/HDFS-12942 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch, > HDFS-12942.003.patch > > > FSDataSetImpl#moveBlock works in following following 3 steps: > # first creates a new replicaInfo object > # calls finalizeReplica to finalize it. > # Calls removeOldReplica to remove oldReplica. > A client can potentially append to the old replica between step 1 and 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-12942: -- Attachment: HDFS-12942.003.patch > Synchronization issue in FSDataSetImpl#moveBlock > > > Key: HDFS-12942 > URL: https://issues.apache.org/jira/browse/HDFS-12942 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch, > HDFS-12942.003.patch > > > FSDataSetImpl#moveBlock works in following following 3 steps: > # first creates a new replicaInfo object > # calls finalizeReplica to finalize it. > # Calls removeOldReplica to remove oldReplica. > A client can potentially append to the old replica between step 1 and 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12812) Ozone: dozone: initialize scm directory on cluster startup
[
https://issues.apache.org/jira/browse/HDFS-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299228#comment-16299228
]
Anu Engineer commented on HDFS-12812:
-
KSM is not running after I apply this patch.
{noformat}
STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r
d3acb451a0c9e89cc44985233066c35d291ec4e8; compiled by 'aengineer' on
2017-12-20T22:52Z
STARTUP_MSG: java = 1.8.0_151
/
2017-12-20 23:11:55 INFO KeySpaceManager:51 - registered UNIX signal handlers
for [TERM, HUP, INT]
2017-12-20 23:11:56 WARN NativeCodeLoader:60 - Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
2017-12-20 23:11:56 ERROR KeySpaceManager:268 - Failed to start the
KeyspaceManager.
org.apache.hadoop.ozone.ksm.exceptions.KSMException: KSM not initialized.
at
org.apache.hadoop.ozone.ksm.KeySpaceManager.(KeySpaceManager.java:134)
at
org.apache.hadoop.ozone.ksm.KeySpaceManager.createKSM(KeySpaceManager.java:307)
at
org.apache.hadoop.ozone.ksm.KeySpaceManager.main(KeySpaceManager.java:262)
2017-12-20 23:11:56 INFO ExitUtil:210 - Exiting with status 1:
org.apache.hadoop.ozone.ksm.exceptions.KSMException: KSM not initialized.
2017-12-20 23:11:56 INFO KeySpaceManager:51 - SHUTDOWN_MSG:
/
SHUTDOWN_MSG: Shutting down KeySpaceManager at 5dfe646f02f1/172.18.0.3
/
{noformat}
I cleared all docker images on my system, pulled new images and this is the
output from from running {{docker logs ozone_ksm_1}}
> Ozone: dozone: initialize scm directory on cluster startup
> --
>
> Key: HDFS-12812
> URL: https://issues.apache.org/jira/browse/HDFS-12812
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Attachments: HADOOP-12812-HDFS-7240.001.patch,
> HDFS-12812-HDFS-7240.002.patch
>
>
> HDFS-12739 fixed the scm, but after the patch it couldn't be started without
> a separated `hdfs scm -init` any more. Unfortunatelly it breaks the
> docker-compose functionality.
> This is handled int the starter script of the base runner docker image for
> namenode. I also fixed this in the runner docker image
> (https://github.com/elek/hadoop/commit/b347eb4bfca37d84dbcdd8c4bf353219d876a9b7)
> will upload the improved patch to the HADOOP-14898.
> In this patch I just add a new environment variable to init the scm if the
> version file doesn't exist.
> To test:
> Do a full build and go to the dev-support/compose/ozone:
> ```
> docker-compose pull
> docker-compose up -d
> ```
> Note: the pull is important as the new docker image -- which could handle the
> new environment variable -- should be used
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12812) Ozone: dozone: initialize scm directory on cluster startup
[
https://issues.apache.org/jira/browse/HDFS-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299202#comment-16299202
]
genericqa commented on HDFS-12812:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 48s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d11161b |
| JIRA Issue | HDFS-12812 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903119/HDFS-12812-HDFS-7240.002.patch
|
| Optional Tests | asflicense |
| uname | Linux 2418c6f5f6ca 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-7240 / d3acb45 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 334 (vs. ulimit of 5000) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/22476/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: dozone: initialize scm directory on cluster startup
> --
>
> Key: HDFS-12812
> URL: https://issues.apache.org/jira/browse/HDFS-12812
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Attachments: HADOOP-12812-HDFS-7240.001.patch,
> HDFS-12812-HDFS-7240.002.patch
>
>
> HDFS-12739 fixed the scm, but after the patch it couldn't be started without
> a separated `hdfs scm -init` any more. Unfortunatelly it breaks the
> docker-compose functionality.
> This is handled int the starter script of the base runner docker image for
> namenode. I also fixed this in the runner docker image
> (https://github.com/elek/hadoop/commit/b347eb4bfca37d84dbcdd8c4bf353219d876a9b7)
> will upload the improved patch to the HADOOP-14898.
> In this patch I just add a new environment variable to init the scm if the
> version file doesn't exist.
> To test:
> Do a full build and go to the dev-support/compose/ozone:
> ```
> docker-compose pull
> docker-compose up -d
> ```
> Note: the pull is important as the new docker image -- which could handle the
> new environment variable -- should be used
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12812) Ozone: dozone: initialize scm directory on cluster startup
[ https://issues.apache.org/jira/browse/HDFS-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299181#comment-16299181 ] Elek, Marton commented on HDFS-12812: - KSM initialization support is added to the patch. Base image at HADOOP-14898 is also updated. Without this patch we can't use docker based ozone for local development. > Ozone: dozone: initialize scm directory on cluster startup > -- > > Key: HDFS-12812 > URL: https://issues.apache.org/jira/browse/HDFS-12812 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HADOOP-12812-HDFS-7240.001.patch, > HDFS-12812-HDFS-7240.002.patch > > > HDFS-12739 fixed the scm, but after the patch it couldn't be started without > a separated `hdfs scm -init` any more. Unfortunatelly it breaks the > docker-compose functionality. > This is handled int the starter script of the base runner docker image for > namenode. I also fixed this in the runner docker image > (https://github.com/elek/hadoop/commit/b347eb4bfca37d84dbcdd8c4bf353219d876a9b7) > will upload the improved patch to the HADOOP-14898. > In this patch I just add a new environment variable to init the scm if the > version file doesn't exist. > To test: > Do a full build and go to the dev-support/compose/ozone: > ``` > docker-compose pull > docker-compose up -d > ``` > Note: the pull is important as the new docker image -- which could handle the > new environment variable -- should be used -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12812) Ozone: dozone: initialize scm directory on cluster startup
[ https://issues.apache.org/jira/browse/HDFS-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDFS-12812: Attachment: HDFS-12812-HDFS-7240.002.patch > Ozone: dozone: initialize scm directory on cluster startup > -- > > Key: HDFS-12812 > URL: https://issues.apache.org/jira/browse/HDFS-12812 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HADOOP-12812-HDFS-7240.001.patch, > HDFS-12812-HDFS-7240.002.patch > > > HDFS-12739 fixed the scm, but after the patch it couldn't be started without > a separated `hdfs scm -init` any more. Unfortunatelly it breaks the > docker-compose functionality. > This is handled int the starter script of the base runner docker image for > namenode. I also fixed this in the runner docker image > (https://github.com/elek/hadoop/commit/b347eb4bfca37d84dbcdd8c4bf353219d876a9b7) > will upload the improved patch to the HADOOP-14898. > In this patch I just add a new environment variable to init the scm if the > version file doesn't exist. > To test: > Do a full build and go to the dev-support/compose/ozone: > ``` > docker-compose pull > docker-compose up -d > ``` > Note: the pull is important as the new docker image -- which could handle the > new environment variable -- should be used -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[
https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299173#comment-16299173
]
Virajith Jalaparti commented on HDFS-12942:
---
To be consistent with the existing code, I don't think you need to use {{this}}
on {{moveBlock}}. Other than that and pending jenkins, patch v2 LGTM.
> Synchronization issue in FSDataSetImpl#moveBlock
>
>
> Key: HDFS-12942
> URL: https://issues.apache.org/jira/browse/HDFS-12942
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch
>
>
> FSDataSetImpl#moveBlock works in following following 3 steps:
> # first creates a new replicaInfo object
> # calls finalizeReplica to finalize it.
> # Calls removeOldReplica to remove oldReplica.
> A client can potentially append to the old replica between step 1 and 2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12942: -- Status: Open (was: Patch Available) > Synchronization issue in FSDataSetImpl#moveBlock > > > Key: HDFS-12942 > URL: https://issues.apache.org/jira/browse/HDFS-12942 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch > > > FSDataSetImpl#moveBlock works in following following 3 steps: > # first creates a new replicaInfo object > # calls finalizeReplica to finalize it. > # Calls removeOldReplica to remove oldReplica. > A client can potentially append to the old replica between step 1 and 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12942: -- Status: Patch Available (was: Open) > Synchronization issue in FSDataSetImpl#moveBlock > > > Key: HDFS-12942 > URL: https://issues.apache.org/jira/browse/HDFS-12942 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch > > > FSDataSetImpl#moveBlock works in following following 3 steps: > # first creates a new replicaInfo object > # calls finalizeReplica to finalize it. > # Calls removeOldReplica to remove oldReplica. > A client can potentially append to the old replica between step 1 and 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[
https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299112#comment-16299112
]
genericqa commented on HDFS-12942:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 46s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
21s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 9s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 122 unchanged - 0 fixed = 123 total (was 122) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 16m
32s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
32s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
27s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 29s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
39s{color} | {color:red} The patch generated 4 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12942 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903092/HDFS-12942.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux ecb110255ffa 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1ba491ff |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/22475/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/22475/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/22475/artifact/out/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
|
| javadoc |
https://builds.apache.org/job/Pre
[jira] [Commented] (HDFS-12945) Switch to ClientProtocol instead of NamenodeProtocols in NamenodeWebHdfsMethods
[
https://issues.apache.org/jira/browse/HDFS-12945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299051#comment-16299051
]
genericqa commented on HDFS-12945:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
41s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 117 unchanged - 3 fixed = 117 total (was 120) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 25s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}125m 28s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}186m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12945 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903078/HDFS-12945.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux bcde494e1541 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 13ad747 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/22474/artifa
[jira] [Updated] (HDFS-12863) Ozone: Avoid configuration.get() call for each container report
[
https://issues.apache.org/jira/browse/HDFS-12863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-12863:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-7240
Status: Resolved (was: Patch Available)
[~elek] Thanks for comments. [~ajisakaa] Thanks for the contribution. I have
committed this to the feature branch.
> Ozone: Avoid configuration.get() call for each container report
> ---
>
> Key: HDFS-12863
> URL: https://issues.apache.org/jira/browse/HDFS-12863
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Minor
> Fix For: HDFS-7240
>
> Attachments: HDFS-12863-HDFS-7240.001.patch,
> HDFS-12863-HDFS-7240.002.patch
>
>
> In ContainerReportManagerImpl.java,
> {{OzoneClientUtils.getScmHeartbeatInterval(config)}} is called for each
> container report. We should cache the interval and avoid calling
> {{Configuration.get(name)}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12938) TestErasureCodigCLI testAll failing consistently.
[
https://issues.apache.org/jira/browse/HDFS-12938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299028#comment-16299028
]
Xiao Chen commented on HDFS-12938:
--
Seems HDFS-12819's precommit didn't run this test...
Thanks Ajay for the quick fix. There's another change s/an non-empty/a
non-empty/. +1 pending this and precommit.
> TestErasureCodigCLI testAll failing consistently.
> -
>
> Key: HDFS-12938
> URL: https://issues.apache.org/jira/browse/HDFS-12938
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, hdfs
>Affects Versions: 3.1.0
>Reporter: Rushabh S Shah
>Assignee: Ajay Kumar
> Attachments: HDFS-12938.001.patch
>
>
> {{TestErasureCodingCLI#testAll}} is failing consistently.
> It failed in this precommit:
> https://builds.apache.org/job/PreCommit-HDFS-Build/22435/testReport/org.apache.hadoop.cli/TestErasureCodingCLI/testAll/
> I ran locally on my laptop and it failed too.
> It failed with this stack trace:
> {noformat}
> java.lang.AssertionError: One of the tests failed. See the Detailed results
> to identify the command that failed
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.cli.CLITestHelper.displayResults(CLITestHelper.java:264)
> at org.apache.hadoop.cli.CLITestHelper.tearDown(CLITestHelper.java:126)
> at
> org.apache.hadoop.cli.TestErasureCodingCLI.tearDown(TestErasureCodingCLI.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> Below is the detailed report from
> {{org.apache.hadoop.cli.TestErasureCodingCLI-output.txt}}.
> {noformat}
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(156)) -
> ---
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(157)) - Test ID: [15]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(158)) -Test Description:
> [setPolicy : set policy on non-empty directory]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(159)) -
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -mkdir /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -touchz /ecdir/file1]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -setPolicy -policy RS-6-3-1024k -path /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(167)) -
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(170)) -Cleanup Commands: [-fs
> hdfs://localhost:52345 -rm -R /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(174)) -
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(178)) - Comparator:
> [SubstringComparator]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(180)) - Comparision result:
> [fail]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(182)) - Expected output:
> [Warning: setting erasure coding policy on an non-empty directory will not
> automatically convert existing data to RS-6-3-1024]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(184)) - Actual output:
> [Set erasure coding policy RS-6-3-1024k on /ecdir
> Warning: setting erasure coding po
[jira] [Updated] (HDFS-12726) BlockPlacementPolicyDefault's debugLoggingBuilder may not be logged
[
https://issues.apache.org/jira/browse/HDFS-12726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-12726:
-
Resolution: Duplicate
Status: Resolved (was: Patch Available)
Resolved as a dup, let's follow up on HDFS-9023. Thanks Surendra.
> BlockPlacementPolicyDefault's debugLoggingBuilder may not be logged
> ---
>
> Key: HDFS-12726
> URL: https://issues.apache.org/jira/browse/HDFS-12726
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: logging
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Labels: supportability
> Attachments: HDFS-12726.01.patch
>
>
> During debugging HDFS-12725, {{BlockPlacementPolicyDefault's}} class'
> {{debugLoggingBuilder}} does a lot of {{get}} and {{append}}, but never
> {{toString}} and {{LOG.debug}}'ed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9023) When NN is not able to identify DN for replication, reason behind it can be logged
[
https://issues.apache.org/jira/browse/HDFS-9023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-9023:
Attachment: HDFS-9023.02.patch
> When NN is not able to identify DN for replication, reason behind it can be
> logged
> --
>
> Key: HDFS-9023
> URL: https://issues.apache.org/jira/browse/HDFS-9023
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, namenode
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HDFS-9023.01.patch, HDFS-9023.02.patch
>
>
> When NN is not able to identify DN for replication, reason behind it can be
> logged (at least critical information why DNs not chosen like disk is full).
> At present it is expected to enable debug log.
> For example the reason for below error looks like all 7 DNs are busy for data
> writes. But at client or NN side no hint is given in the log message.
> {noformat}
> File /tmp/logs/spark/logs/application_1437051383180_0610/xyz-195_26009.tmp
> could only be replicated to 0 nodes instead of minReplication (=1). There
> are 7 datanode(s) running and no node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1553)
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9023) When NN is not able to identify DN for replication, reason behind it can be logged
[
https://issues.apache.org/jira/browse/HDFS-9023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-9023:
Attachment: HDFS-9023.01.patch
> When NN is not able to identify DN for replication, reason behind it can be
> logged
> --
>
> Key: HDFS-9023
> URL: https://issues.apache.org/jira/browse/HDFS-9023
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, namenode
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HDFS-9023.01.patch, HDFS-9023.02.patch
>
>
> When NN is not able to identify DN for replication, reason behind it can be
> logged (at least critical information why DNs not chosen like disk is full).
> At present it is expected to enable debug log.
> For example the reason for below error looks like all 7 DNs are busy for data
> writes. But at client or NN side no hint is given in the log message.
> {noformat}
> File /tmp/logs/spark/logs/application_1437051383180_0610/xyz-195_26009.tmp
> could only be replicated to 0 nodes instead of minReplication (=1). There
> are 7 datanode(s) running and no node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1553)
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9023) When NN is not able to identify DN for replication, reason behind it can be logged
[
https://issues.apache.org/jira/browse/HDFS-9023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299006#comment-16299006
]
Xiao Chen commented on HDFS-9023:
-
Thanks a lot [~surendrasingh] for the review and reassignment.
I resolved HDFS-12726 as a dup since this jira is created earlier. Copying your
comments over to continue here.
{quote}
# {{private static final ThreadLocal chooseRandomReasons}}
Here HashMap is a raw type. it should be parameterized.
e.x :
{code}
private static final ThreadLocal>
chooseRandomReasons = ThreadLocal
.withInitial(() -> new HashMap<>());
{code}
# No need to log failure reason here.
{code}
Map reasonMap = chooseRandomReasons.get();
if (!reasonMap.isEmpty()) {
LOG.info("Not enough replicas was chosen. Reason:{}", reasonMap);
}
throw new NotEnoughReplicasException(detail);
{code}
just append reason in {{detail}} string. {{NotEnoughReplicasException}} message
will be logged in {{chooseTarget(..)}} method.
{code}
if (LOG.isTraceEnabled()) {
LOG.trace(message, e);
} else {
LOG.warn(message + " " + e.getMessage());
}
{code}
# I feel {{NodeNotChosenReason#NO_GOOD_STORAGE}} should be
{{NodeNotChosenReason#NOTE_ENOUGH_STORAGE_SPACE}}. This we are using when
{{(requiredSize > remaining - scheduledSize)}}
{quote}
bq. 1.
good catch, fixed
bq. 2. append reason in detail string.
I though about this too, but did not change the behavior because It seems
{{badTarget}}'s filtering logic is very explicit.
I git blamed to [the earliest available version on
github|https://github.com/apache/hadoop/blob/a196766ea07775f18ded69bd9e8d239f8cfd3ccc/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockPlacementPolicyDefault.java#L413],
but the exception message filtering is still there. My guess would be we don't
want a benign (or, non critical) exception to have a message that's too long.
I'd like to keep the day 0 behavior to prevent surprise.
bq. 3. NOT_ENOUGH_STORAGE_SPACE
done. Added the storage type to the message to make it clear that the space is
specific to the type.
patch 2 also addressed 2 other issues I found:
- {{DatanodeDescriptor}}'s log added by HDFS-11494 should use the BPP logger.
This is the original behavior before HDFS-8946. (This can't be changed to use
{{BPPD#logNodeIsNotChosen}} because it may also be called by
{{chooseLocalStorage}}, which doesn't go through the logging in {{chooseRandom}}
- only log when the stringbuilder has more than "[".
Some of the tests I ran to look at the logs are:
{{TestErasureCodingMultipleRacks}}, {{TestDefaultBlockPlacementPolicy}},
{{TestNamenodeStorageDirectives}}.
> When NN is not able to identify DN for replication, reason behind it can be
> logged
> --
>
> Key: HDFS-9023
> URL: https://issues.apache.org/jira/browse/HDFS-9023
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, namenode
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Xiao Chen
>Priority: Critical
>
> When NN is not able to identify DN for replication, reason behind it can be
> logged (at least critical information why DNs not chosen like disk is full).
> At present it is expected to enable debug log.
> For example the reason for below error looks like all 7 DNs are busy for data
> writes. But at client or NN side no hint is given in the log message.
> {noformat}
> File /tmp/logs/spark/logs/application_1437051383180_0610/xyz-195_26009.tmp
> could only be replicated to 0 nodes instead of minReplication (=1). There
> are 7 datanode(s) running and no node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1553)
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298996#comment-16298996 ] Ajay Kumar edited comment on HDFS-12942 at 12/20/17 7:55 PM: - [~virajith], Thanks for review. Addressed your suggestions in patch v2. Also addressed checkstyle issues from jenkins. [~arpitagarwal], thanks for suggestion on cleanup of old replica. was (Author: ajayydv): [~virajith], Thanks for review. Addressed your suggestions in patch v2. Also addressed checkstyle issues from jenkins. > Synchronization issue in FSDataSetImpl#moveBlock > > > Key: HDFS-12942 > URL: https://issues.apache.org/jira/browse/HDFS-12942 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch > > > FSDataSetImpl#moveBlock works in following following 3 steps: > # first creates a new replicaInfo object > # calls finalizeReplica to finalize it. > # Calls removeOldReplica to remove oldReplica. > A client can potentially append to the old replica between step 1 and 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-12942: -- Attachment: HDFS-12942.002.patch [~virajith], Thanks for review. Addressed your suggestions in patch v2. Also addressed checkstyle issues from jenkins. > Synchronization issue in FSDataSetImpl#moveBlock > > > Key: HDFS-12942 > URL: https://issues.apache.org/jira/browse/HDFS-12942 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch > > > FSDataSetImpl#moveBlock works in following following 3 steps: > # first creates a new replicaInfo object > # calls finalizeReplica to finalize it. > # Calls removeOldReplica to remove oldReplica. > A client can potentially append to the old replica between step 1 and 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10419) Building HDFS on top of Ozone's storage containers
[ https://issues.apache.org/jira/browse/HDFS-10419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298979#comment-16298979 ] Joe Pallas commented on HDFS-10419: --- So the consensus mechanism guarantees all the container replicas are consistent, but the NN is not part of that consensus. Since the container reports don't have block-level information, the NN will make an explicit request to one of the container replicas for block info in the case of a client failure. Is that correct? > Building HDFS on top of Ozone's storage containers > -- > > Key: HDFS-10419 > URL: https://issues.apache.org/jira/browse/HDFS-10419 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: Evolving NN using new block-container layer.pdf > > > In HDFS-7240, Ozone defines storage containers to store both the data and the > metadata. The storage container layer provides an object storage interface > and aims to manage data/metadata in a distributed manner. More details about > storage containers can be found in the design doc in HDFS-7240. > HDFS can adopt the storage containers to store and manage blocks. The general > idea is: > # Each block can be treated as an object and the block ID is the object's key. > # Blocks will still be stored in DataNodes but as objects in storage > containers. > # The block management work can be separated out of the NameNode and will be > handled by the storage container layer in a more distributed way. The > NameNode will only manage the namespace (i.e., files and directories). > # For each file, the NameNode only needs to record a list of block IDs which > are used as keys to obtain real data from storage containers. > # A new DFSClient implementation talks to both NameNode and the storage > container layer to read/write. > HDFS, especially the NameNode, can get much better scalability from this > design. Currently the NameNode's heaviest workload comes from the block > management, which includes maintaining the block-DataNode mapping, receiving > full/incremental block reports, tracking block states (under/over/miss > replicated), and joining every writing pipeline protocol to guarantee the > data consistency. These work bring high memory footprint and make NameNode > suffer from GC. HDFS-5477 already proposes to convert BlockManager as a > service. If we can build HDFS on top of the storage container layer, we not > only separate out the BlockManager from the NameNode, but also replace it > with a new distributed management scheme. > The storage container work is currently in progress in HDFS-7240, and the > work proposed here is still in an experimental/exploring stage. We can do > this experiment in a feature branch so that people with interests can be > involved. > A design doc will be uploaded later explaining more details. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12751) Ozone: SCM: update container allocated size to container db for all the open containers in ContainerStateManager#close
[
https://issues.apache.org/jira/browse/HDFS-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298970#comment-16298970
]
Chen Liang commented on HDFS-12751:
---
Thanks [~nandakumar131] for the comment!
bq. 1. Getting all the containers from ContainerStateManager for updating
allocated bytes.
Thanks for point this out, especially the link the JIRAs. Then let's this
revisit this when HDFS-12522 is done, For now I think this should be fine as
this should affect the time it takes to shut down SCM. I will follow up JIRAs
later.
bq. 2. Exposing setter method for allocated bytes - setAllocatedBytes
Thanks for bringing up this point. I wonder though, is there a code path where
SCM will read and rely on {{ContainerInfo}} instance from client? I was under
impression that client only reads ContaienrInfo, but whatever the client does
with this object will not be taken by the server side.
> Ozone: SCM: update container allocated size to container db for all the open
> containers in ContainerStateManager#close
> --
>
> Key: HDFS-12751
> URL: https://issues.apache.org/jira/browse/HDFS-12751
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: Chen Liang
> Fix For: HDFS-7240
>
> Attachments: HDFS-12751-HDFS-7240.001.patch,
> HDFS-12751-HDFS-7240.002.patch
>
>
> Container allocated size is maintained in memory by
> {{ContainerStateManager}}, this has to be updated in container db when we
> shutdown SCM. {{ContainerStateManager#close}} will be called during SCM
> shutdown, so updating allocated size for all the open containers should be
> done here.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[
https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298948#comment-16298948
]
Virajith Jalaparti commented on HDFS-12942:
---
Hi [~ajayydv], thanks for posting the patch to fix this. A few comments:
# Is it necessary to call {{processBlockInvalidation}} in
{{cleanupNewReplica}}? The {{newReplicaInfo}} passed in isn't used for reading
as it is just temporary for the block move. This call will invalidate any
references to the original replica which was used to create the
newReplicaInfo}}. So, I think this call is not required.
# The rest of the code in {{cleanupNewReplica}} is same as the deletion logic
in {{removeOldReplica}}. If this method is being added, it would be useful to
refactor {{removeOldReplica}} and call {{cleanupNewReplica}} instead of having
the same logic in two places. I would rename {{cleanupNewReplica}} to say
{{cleanupReplica}} when this refactoring is done.
# I think a method name like {{FsDatasetImpl#copyReplicaToVolume}} instead of
{{FsDatasetImpl#createNewReplicaObj}} would make it clearer what that method is
doing. I would also clarify in the javadoc that the method creates a
{{TEMPORARY}} replica in the volume passed in.
# A javadoc for {{TestFsDatasetImpl#createNewReplicaObj}} would be useful.
# {{"Generation Stamp should be monotonically increased. That assumption is
violated here."}} I think the latter part of the message ({{"That assumption is
violated here"}}) isn't required.
> Synchronization issue in FSDataSetImpl#moveBlock
>
>
> Key: HDFS-12942
> URL: https://issues.apache.org/jira/browse/HDFS-12942
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12942.001.patch
>
>
> FSDataSetImpl#moveBlock works in following following 3 steps:
> # first creates a new replicaInfo object
> # calls finalizeReplica to finalize it.
> # Calls removeOldReplica to remove oldReplica.
> A client can potentially append to the old replica between step 1 and 2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12938) TestErasureCodigCLI testAll failing consistently.
[
https://issues.apache.org/jira/browse/HDFS-12938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HDFS-12938:
--
Attachment: HDFS-12938.001.patch
Seems there is difference in warning message coming from {{ECAdmin}} and one
we are expecting in {{testErasureCodingConf.xml}}. Uploading patch for review.
> TestErasureCodigCLI testAll failing consistently.
> -
>
> Key: HDFS-12938
> URL: https://issues.apache.org/jira/browse/HDFS-12938
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, hdfs
>Affects Versions: 3.1.0
>Reporter: Rushabh S Shah
>Assignee: Ajay Kumar
> Attachments: HDFS-12938.001.patch
>
>
> {{TestErasureCodingCLI#testAll}} is failing consistently.
> It failed in this precommit:
> https://builds.apache.org/job/PreCommit-HDFS-Build/22435/testReport/org.apache.hadoop.cli/TestErasureCodingCLI/testAll/
> I ran locally on my laptop and it failed too.
> It failed with this stack trace:
> {noformat}
> java.lang.AssertionError: One of the tests failed. See the Detailed results
> to identify the command that failed
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.cli.CLITestHelper.displayResults(CLITestHelper.java:264)
> at org.apache.hadoop.cli.CLITestHelper.tearDown(CLITestHelper.java:126)
> at
> org.apache.hadoop.cli.TestErasureCodingCLI.tearDown(TestErasureCodingCLI.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> Below is the detailed report from
> {{org.apache.hadoop.cli.TestErasureCodingCLI-output.txt}}.
> {noformat}
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(156)) -
> ---
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(157)) - Test ID: [15]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(158)) -Test Description:
> [setPolicy : set policy on non-empty directory]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(159)) -
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -mkdir /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -touchz /ecdir/file1]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -setPolicy -policy RS-6-3-1024k -path /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(167)) -
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(170)) -Cleanup Commands: [-fs
> hdfs://localhost:52345 -rm -R /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(174)) -
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(178)) - Comparator:
> [SubstringComparator]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(180)) - Comparision result:
> [fail]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(182)) - Expected output:
> [Warning: setting erasure coding policy on an non-empty directory will not
> automatically convert existing data to RS-6-3-1024]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(184)) - Actual output:
> [Set erasure coding policy RS-6-3-1024k on /ecdir
> Warning: setting erasure coding policy on a non-empty directory
[jira] [Commented] (HDFS-12863) Ozone: Avoid configuration.get() call for each container report
[
https://issues.apache.org/jira/browse/HDFS-12863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298913#comment-16298913
]
Anu Engineer commented on HDFS-12863:
-
+1, thanks for the contribution. I will commit this to the feature branch.
> Ozone: Avoid configuration.get() call for each container report
> ---
>
> Key: HDFS-12863
> URL: https://issues.apache.org/jira/browse/HDFS-12863
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Minor
> Attachments: HDFS-12863-HDFS-7240.001.patch,
> HDFS-12863-HDFS-7240.002.patch
>
>
> In ContainerReportManagerImpl.java,
> {{OzoneClientUtils.getScmHeartbeatInterval(config)}} is called for each
> container report. We should cache the interval and avoid calling
> {{Configuration.get(name)}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12908) Ozone: write chunk call fails because of Metrics registry exception
[
https://issues.apache.org/jira/browse/HDFS-12908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298907#comment-16298907
]
Nanda kumar commented on HDFS-12908:
Mukul, the patch is not applying anymore. Please update it on top of latest
changes.
> Ozone: write chunk call fails because of Metrics registry exception
> ---
>
> Key: HDFS-12908
> URL: https://issues.apache.org/jira/browse/HDFS-12908
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: HDFS-7240
>
> Attachments: HDFS-12908-HDFS-7240.001.patch,
> HDFS-12908-HDFS-7240.002.patch, HDFS-12908-HDFS-7240.003.patch
>
>
> write chunk call fail because of Metric registration exception.
> {code}
> 2017-12-08 04:02:19,894 WARN org.apache.hadoop.metrics2.util.MBeans: Error
> creating MBean object name:
> Hadoop:service=Ozone,name=RocksDbStore,dbName=container.db
> org.apache.hadoop.metrics2.MetricsException:
> org.apache.hadoop.metrics2.MetricsException:
> Hadoop:service=Ozone,name=RocksDbStore,dbName=container.db already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:135)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(DefaultMetricsSystem.java:110)
> at
> org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:155)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:87)
> at org.apache.hadoop.utils.RocksDBStore.(RocksDBStore.java:77)
> at
> org.apache.hadoop.utils.MetadataStoreBuilder.build(MetadataStoreBuilder.java:115)
> at
> org.apache.hadoop.ozone.container.common.utils.ContainerCache.getDB(ContainerCache.java:138)
> at
> org.apache.hadoop.ozone.container.common.helpers.KeyUtils.getDB(KeyUtils.java:65)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.readContainerInfo(ContainerManagerImpl.java:261)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.createContainer(ContainerManagerImpl.java:330)
> at
> org.apache.hadoop.ozone.container.common.impl.Dispatcher.handleCreateContainer(Dispatcher.java:399)
> at
> org.apache.hadoop.ozone.container.common.impl.Dispatcher.containerProcessHandler(Dispatcher.java:158)
> at
> org.apache.hadoop.ozone.container.common.impl.Dispatcher.dispatch(Dispatcher.java:105)
> at
> org.apache.hadoop.ozone.container.common.transport.server.XceiverServerHandler.channelRead0(XceiverServerHandler.java:61)
> at
> org.apache.hadoop.ozone.container.common.transport.server.XceiverServerHandler.channelRead0(XceiverServerHandler.java:32)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> at
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1302)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> a
[jira] [Commented] (HDFS-12943) Consistent Reads from Standby Node
[
https://issues.apache.org/jira/browse/HDFS-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298876#comment-16298876
]
Chao Sun commented on HDFS-12943:
-
bq. Thanks Chao Sun, interesting results! You used only 1 SBN to server reads
right?
Yes I used 1 ANN + 1SBN + 1ONN (observer NN).
bq. In both configurations (with and without stale reads), I assume you were
saturating the system?
In the stale read case, the RPC queue time on the ANN was less than 5ms, while
on ONN it was between 0 to 30ms. In the non-stale read case, the RPC queue time
on ANN was around 130-140ms. So I guess the ANN was not saturated when stale
read is enabled?
bq. Did you use Namesystem unfair locking?
The ANN didn't use unfair locking. The ONN used unfair locking + async audit
logging (we have an internal patch to use log4j 2.x) + async edit logging. Do
you think it will make a difference if unfair locking is used on ANN?
bq. If I understand correctly, both your test and the Dynamometer test are more
like trace-driven micro benchmarks, where a container issues a certain type of
RPC at given timestamp. Chris was probably referring to a test job with "real
code" like if !file_exists(path) then create_file(path), where the blocking
relationship between calls are miniced.
Yes the test was pretty simple. It is basically:
{code}
loop {
x = randInt(0, 100)
if (x < 6) {
fs.createNewFile(..)
fs.rename(..)
fs.delete(..)
} else if (x < 10) {
fs.listStatus(..)
} else if (x < 40) {
fs.getFileBlockLocations(..)
} else {
fs.getFileStatus(..)
}
}
{code}
The file listing was done on a directory with 2K files.
Let me know if you have any suggestion on improving this. It's pretty easy to
change the code and re-run the benchmark.
> Consistent Reads from Standby Node
> --
>
> Key: HDFS-12943
> URL: https://issues.apache.org/jira/browse/HDFS-12943
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: hdfs
>Reporter: Konstantin Shvachko
> Attachments: ConsistentReadsFromStandbyNode.pdf
>
>
> StandbyNode in HDFS is a replica of the active NameNode. The states of the
> NameNodes are coordinated via the journal. It is natural to consider
> StandbyNode as a read-only replica. As with any replicated distributed system
> the problem of stale reads should be resolved. Our main goal is to provide
> reads from standby in a consistent way in order to enable a wide range of
> existing applications running on top of HDFS.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12951) Incorrect javadoc in SaslDataTransferServer.java#receive
[
https://issues.apache.org/jira/browse/HDFS-12951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298844#comment-16298844
]
genericqa commented on HDFS-12951:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
52s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}114m 41s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}165m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.cli.TestErasureCodingCLI |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12951 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903069/HDFS-12951.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux f4aa63b22652 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d62932c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/22473/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22473/artifact/out/patch-un
[jira] [Updated] (HDFS-12950) oiv will fail in secure cluster
[
https://issues.apache.org/jira/browse/HDFS-12950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12950:
Description:
if we execute ls, it will throw following.
{noformat}
hdfs dfs -ls webhdfs://127.0.0.1:5978/
ls: Invalid value for webhdfs parameter "op"
{noformat}
When client is configured with security (i.e "hadoop.security.authentication=
KERBEROS) ,
then webhdfs will request getdelegation token which is not implemented and
hence it will throw “ls: Invalid value for webhdfs parameter "op"”.
was:
We we execute ls, it will throw following.
{noformat}
hdfs dfs -ls webhdfs://127.0.0.1:5978/
ls: Invalid value for webhdfs parameter "op"
{noformat}
When client is configured with security (i.e "hadoop.security.authentication=
KERBEROS) ,
then webhdfs will request getdelegation token which is not implemented and
hence it will throw “ls: Invalid value for webhdfs parameter "op"”.
> oiv will fail in secure cluster
>
>
> Key: HDFS-12950
> URL: https://issues.apache.org/jira/browse/HDFS-12950
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>
> if we execute ls, it will throw following.
> {noformat}
> hdfs dfs -ls webhdfs://127.0.0.1:5978/
> ls: Invalid value for webhdfs parameter "op"
> {noformat}
> When client is configured with security (i.e "hadoop.security.authentication=
> KERBEROS) ,
> then webhdfs will request getdelegation token which is not implemented and
> hence it will throw “ls: Invalid value for webhdfs parameter "op"”.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12932) Fix confusing LOG message for block replication
[
https://issues.apache.org/jira/browse/HDFS-12932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298819#comment-16298819
]
Chao Sun commented on HDFS-12932:
-
Thanks [~vagarychen] and [~ywskycn] for reviewing the patch!
> Fix confusing LOG message for block replication
> ---
>
> Key: HDFS-12932
> URL: https://issues.apache.org/jira/browse/HDFS-12932
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 2.8.3
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HDFS-12932.0.patch, HDFS-12932.1.patch
>
>
> In our cluster we see large number of log messages such as the following:
> {code}
> 2017-12-15 22:55:54,603 INFO
> org.apache.hadoop.hdfs.server.namenode.FSDirectory: Increasing replication
> from 3 to 3 for
> {code}
> This is a little confusing since "from 3 to 3" is not "increasing". Digging
> into it, it seems related to this piece of code:
> {code}
> if (oldBR != -1) {
> if (oldBR > targetReplication) {
> FSDirectory.LOG.info("Decreasing replication from {} to {} for {}",
> oldBR, targetReplication, iip.getPath());
> } else {
> FSDirectory.LOG.info("Increasing replication from {} to {} for {}",
> oldBR, targetReplication, iip.getPath());
> }
> }
> {code}
> Perhaps a {{oldBR == targetReplication}} case is missing?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12945) Switch to ClientProtocol instead of NamenodeProtocols in NamenodeWebHdfsMethods
[ https://issues.apache.org/jira/browse/HDFS-12945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated HDFS-12945: --- Attachment: HDFS-12945.001.patch Fix two checkstyle issues. > Switch to ClientProtocol instead of NamenodeProtocols in > NamenodeWebHdfsMethods > --- > > Key: HDFS-12945 > URL: https://issues.apache.org/jira/browse/HDFS-12945 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Minor > Attachments: HDFS-12945.000.patch, HDFS-12945.001.patch > > > In HDFS-12512 which is to add WebHDFS support to Router-based Federation, we > found it would be good to switch from NamenodeProtocols to ClientProtocol in > NamenodeWebHdfsMethods, to make code sharable between NameNode WebHDFS and > Router WebHDFS. Would like to get some feedbacks about this refactor, any > concerns? > cc [~elgoiri] [~szetszwo] [~sanjay.radia] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12932) Fix confusing LOG message for block replication
[
https://issues.apache.org/jira/browse/HDFS-12932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298764#comment-16298764
]
Hudson commented on HDFS-12932:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13409 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13409/])
HDFS-12932. Fix confusing LOG message for block replication. Contributed (weiy:
rev a78db9919065d06ced819530f44cc7369857)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java
> Fix confusing LOG message for block replication
> ---
>
> Key: HDFS-12932
> URL: https://issues.apache.org/jira/browse/HDFS-12932
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 2.8.3
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HDFS-12932.0.patch, HDFS-12932.1.patch
>
>
> In our cluster we see large number of log messages such as the following:
> {code}
> 2017-12-15 22:55:54,603 INFO
> org.apache.hadoop.hdfs.server.namenode.FSDirectory: Increasing replication
> from 3 to 3 for
> {code}
> This is a little confusing since "from 3 to 3" is not "increasing". Digging
> into it, it seems related to this piece of code:
> {code}
> if (oldBR != -1) {
> if (oldBR > targetReplication) {
> FSDirectory.LOG.info("Decreasing replication from {} to {} for {}",
> oldBR, targetReplication, iip.getPath());
> } else {
> FSDirectory.LOG.info("Increasing replication from {} to {} for {}",
> oldBR, targetReplication, iip.getPath());
> }
> }
> {code}
> Perhaps a {{oldBR == targetReplication}} case is missing?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12932) Fix confusing LOG message for block replication
[
https://issues.apache.org/jira/browse/HDFS-12932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan updated HDFS-12932:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks for the contribution [~csun]
> Fix confusing LOG message for block replication
> ---
>
> Key: HDFS-12932
> URL: https://issues.apache.org/jira/browse/HDFS-12932
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 2.8.3
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HDFS-12932.0.patch, HDFS-12932.1.patch
>
>
> In our cluster we see large number of log messages such as the following:
> {code}
> 2017-12-15 22:55:54,603 INFO
> org.apache.hadoop.hdfs.server.namenode.FSDirectory: Increasing replication
> from 3 to 3 for
> {code}
> This is a little confusing since "from 3 to 3" is not "increasing". Digging
> into it, it seems related to this piece of code:
> {code}
> if (oldBR != -1) {
> if (oldBR > targetReplication) {
> FSDirectory.LOG.info("Decreasing replication from {} to {} for {}",
> oldBR, targetReplication, iip.getPath());
> } else {
> FSDirectory.LOG.info("Increasing replication from {} to {} for {}",
> oldBR, targetReplication, iip.getPath());
> }
> }
> {code}
> Perhaps a {{oldBR == targetReplication}} case is missing?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12932) Fix confusing LOG message for block replication
[
https://issues.apache.org/jira/browse/HDFS-12932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan updated HDFS-12932:
---
Fix Version/s: 3.1.0
> Fix confusing LOG message for block replication
> ---
>
> Key: HDFS-12932
> URL: https://issues.apache.org/jira/browse/HDFS-12932
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 2.8.3
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HDFS-12932.0.patch, HDFS-12932.1.patch
>
>
> In our cluster we see large number of log messages such as the following:
> {code}
> 2017-12-15 22:55:54,603 INFO
> org.apache.hadoop.hdfs.server.namenode.FSDirectory: Increasing replication
> from 3 to 3 for
> {code}
> This is a little confusing since "from 3 to 3" is not "increasing". Digging
> into it, it seems related to this piece of code:
> {code}
> if (oldBR != -1) {
> if (oldBR > targetReplication) {
> FSDirectory.LOG.info("Decreasing replication from {} to {} for {}",
> oldBR, targetReplication, iip.getPath());
> } else {
> FSDirectory.LOG.info("Increasing replication from {} to {} for {}",
> oldBR, targetReplication, iip.getPath());
> }
> }
> {code}
> Perhaps a {{oldBR == targetReplication}} case is missing?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12952) Ozone: Change OzoneFS's semtics to allow readers to see file content while being written
[ https://issues.apache.org/jira/browse/HDFS-12952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-12952: Summary: Ozone: Change OzoneFS's semtics to allow readers to see file content while being written (was: Change OzoneFS's semtics to allow readers to see file content while being written) > Ozone: Change OzoneFS's semtics to allow readers to see file content while > being written > > > Key: HDFS-12952 > URL: https://issues.apache.org/jira/browse/HDFS-12952 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Sanjay Radia >Assignee: Anu Engineer > > Currently Ozone KSM give visibility to a file only when the file is closed, > which is similar to S3 FS. OzoneFs should allow partial file visibility as a > file is being written to match HDFS. > Note this should be fairly straightforward because the block-container layer > maintains block length consistency as a block is being written even under > failures due to its use of Raft. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12952) Change OzoneFS's semtics to allow readers to see file content while being written
Sanjay Radia created HDFS-12952: --- Summary: Change OzoneFS's semtics to allow readers to see file content while being written Key: HDFS-12952 URL: https://issues.apache.org/jira/browse/HDFS-12952 Project: Hadoop HDFS Issue Type: Bug Reporter: Sanjay Radia Assignee: Anu Engineer Currently Ozone KSM give visibility to a file only when the file is closed, which is similar to S3 FS. OzoneFs should allow partial file visibility as a file is being written to match HDFS. Note this should be fairly straightforward because the block-container layer maintains block length consistency as a block is being written even under failures due to its use of Raft. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12932) Fix confusing LOG message for block replication
[
https://issues.apache.org/jira/browse/HDFS-12932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan updated HDFS-12932:
---
Summary: Fix confusing LOG message for block replication (was: Confusing
LOG message for block replication)
> Fix confusing LOG message for block replication
> ---
>
> Key: HDFS-12932
> URL: https://issues.apache.org/jira/browse/HDFS-12932
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 2.8.3
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Minor
> Attachments: HDFS-12932.0.patch, HDFS-12932.1.patch
>
>
> In our cluster we see large number of log messages such as the following:
> {code}
> 2017-12-15 22:55:54,603 INFO
> org.apache.hadoop.hdfs.server.namenode.FSDirectory: Increasing replication
> from 3 to 3 for
> {code}
> This is a little confusing since "from 3 to 3" is not "increasing". Digging
> into it, it seems related to this piece of code:
> {code}
> if (oldBR != -1) {
> if (oldBR > targetReplication) {
> FSDirectory.LOG.info("Decreasing replication from {} to {} for {}",
> oldBR, targetReplication, iip.getPath());
> } else {
> FSDirectory.LOG.info("Increasing replication from {} to {} for {}",
> oldBR, targetReplication, iip.getPath());
> }
> }
> {code}
> Perhaps a {{oldBR == targetReplication}} case is missing?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10419) Building HDFS on top of Ozone's storage containers
[ https://issues.apache.org/jira/browse/HDFS-10419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298728#comment-16298728 ] Sanjay Radia commented on HDFS-10419: - The new block-container layer allows partial visibility of a block even before the block has finalized an closed. It maintains block length consistency without the help of the namespace layer (KSM or NN) as a block is being written even under failures due to its use of Raft. Hence the NN, when plugged into the new block-container layer, will be fine - we can get the HDFS semantics. You are right that the Ozone KSM give visibility to a file only when the file is closed, which is similar to S3 FS. I will create a jira to fix that so that the OzoneFs matches HDFS for partail file visibility as a file is being written. > Building HDFS on top of Ozone's storage containers > -- > > Key: HDFS-10419 > URL: https://issues.apache.org/jira/browse/HDFS-10419 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: Evolving NN using new block-container layer.pdf > > > In HDFS-7240, Ozone defines storage containers to store both the data and the > metadata. The storage container layer provides an object storage interface > and aims to manage data/metadata in a distributed manner. More details about > storage containers can be found in the design doc in HDFS-7240. > HDFS can adopt the storage containers to store and manage blocks. The general > idea is: > # Each block can be treated as an object and the block ID is the object's key. > # Blocks will still be stored in DataNodes but as objects in storage > containers. > # The block management work can be separated out of the NameNode and will be > handled by the storage container layer in a more distributed way. The > NameNode will only manage the namespace (i.e., files and directories). > # For each file, the NameNode only needs to record a list of block IDs which > are used as keys to obtain real data from storage containers. > # A new DFSClient implementation talks to both NameNode and the storage > container layer to read/write. > HDFS, especially the NameNode, can get much better scalability from this > design. Currently the NameNode's heaviest workload comes from the block > management, which includes maintaining the block-DataNode mapping, receiving > full/incremental block reports, tracking block states (under/over/miss > replicated), and joining every writing pipeline protocol to guarantee the > data consistency. These work bring high memory footprint and make NameNode > suffer from GC. HDFS-5477 already proposes to convert BlockManager as a > service. If we can build HDFS on top of the storage container layer, we not > only separate out the BlockManager from the NameNode, but also replace it > with a new distributed management scheme. > The storage container work is currently in progress in HDFS-7240, and the > work proposed here is still in an experimental/exploring stage. We can do > this experiment in a feature branch so that people with interests can be > involved. > A design doc will be uploaded later explaining more details. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12945) Switch to ClientProtocol instead of NamenodeProtocols in NamenodeWebHdfsMethods
[ https://issues.apache.org/jira/browse/HDFS-12945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298715#comment-16298715 ] Íñigo Goiri commented on HDFS-12945: The checkstyles were technically already there but it doesn't hurt to fix them. I don't think this changes the behavior and it makes HDFS-12512 much cleaner. Other than fixing the check styles, +1 > Switch to ClientProtocol instead of NamenodeProtocols in > NamenodeWebHdfsMethods > --- > > Key: HDFS-12945 > URL: https://issues.apache.org/jira/browse/HDFS-12945 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Minor > Attachments: HDFS-12945.000.patch > > > In HDFS-12512 which is to add WebHDFS support to Router-based Federation, we > found it would be good to switch from NamenodeProtocols to ClientProtocol in > NamenodeWebHdfsMethods, to make code sharable between NameNode WebHDFS and > Router WebHDFS. Would like to get some feedbacks about this refactor, any > concerns? > cc [~elgoiri] [~szetszwo] [~sanjay.radia] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12934) RBF: Federation supports global quota
[
https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298707#comment-16298707
]
Íñigo Goiri commented on HDFS-12934:
[~linyiqun], thanks for [^RBF support global quota.pdf]; it looks good and it
helped me understand better.
A couple minor things:
* I think we should have flags to enable the quota system at the Router level
(basically making sure we don't run additional services).
* I guess we also want to implement the {{setQuotaSpace()}}.
* I didn't get what you meant about {{ClientProtocol#getQuota()}}; will we
implement it or not?
As you mention, it might not be that big and fit in a single JIRA; feel free to
give it a try.
> RBF: Federation supports global quota
> -
>
> Key: HDFS-12934
> URL: https://issues.apache.org/jira/browse/HDFS-12934
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Labels: RBF
> Attachments: RBF support global quota.pdf
>
>
> Now federation doesn't support set the global quota for each folder.
> Currently the quota will be applied for each subcluster under the specified
> folder via RPC call.
> It will be very useful for users that federation can support setting global
> quota and exposing the command of this.
> In a federated environment, a folder can be spread across multiple
> subclusters. For this reason, we plan to solve this by following way:
> # Set global quota across each subcluster. We don't allow each subcluster can
> exceed maximun quota value.
> # We need to construct one cache map for storing the sum
> quota usage of these subclusters under federation folder. Every time we want
> to do WRITE operation under specified folder, we will get its quota usage
> from cache and verify its quota. If quota exceeded, throw exception,
> otherwise update its quota usage in cache when finishing operations.
> The quota will be set to mount table and as a new field in mount table. The
> set/unset command will be like:
> {noformat}
> hdfs dfsrouteradmin -setQuota -ns -ss
> hdfs dfsrouteradmin -clrQuota
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12911) [SPS]: Fix review comments from discussions in HDFS-10285
[
https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-12911:
Attachment: HDFS-12911-HDFS-10285-01.patch
Thanks a lot [~chris.douglas] for the help and the useful patch. I have tried
an attempt incorporating the {{Context}} interface and separated out the
locking from the {{#analyseBlocksStorageMovementsAndAssignToDN}} function.
Following are the changes in the patch:
# Incorporated {{Context}} interface and {{IntraNNSPSContext}} impl.
# Separated out the querying and assigning block movements so that the lock can
be taken only for the querying logic.
# Moved the following sps classes to new package
{{'org.apache.hadoop.hdfs.server.namenode.sps'}}
- StoragePolicySatisfier
- BlockStorageMovementNeeded
- BlockStorageMovementAttemptedItems
Due to the package changes, the patch becomes quite large. Please apply the
changes to your env and I hope that will help you to understand the proposed
difference easily.
Appreciate your comments!
> [SPS]: Fix review comments from discussions in HDFS-10285
> -
>
> Key: HDFS-12911
> URL: https://issues.apache.org/jira/browse/HDFS-12911
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Rakesh R
> Attachments: HDFS-12911-HDFS-10285-01.patch, HDFS-12911.00.patch
>
>
> This is the JIRA for tracking the possible improvements or issues discussed
> in main JIRA
> So far comments to handle
> Daryn:
> # Lock should not kept while executing placement policy.
> # While starting up the NN, SPS Xattrs checks happen even if feature
> disabled. This could potentially impact the startup speed.
> UMA:
> # I am adding one more possible improvement to reduce Xattr objects
> significantly.
> SPS Xattr is constant object. So, we create one Xattr deduplication object
> once statically and use the same object reference when required to add SPS
> Xattr to Inode. So, here additional bytes required for storing SPS Xattr
> would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr
> overhead should come down significantly IMO. Lets explore the feasibility on
> this option.
> Xattr list Future will not be specially created for SPS, that list would have
> been created by SetStoragePolicy already on the same directory. So, no extra
> Feature creation because of SPS alone.
> # Currently SPS putting long id objects in Q for tracking SPS called Inodes.
> So, it is additional created and size of it would be (obj ref + value) = (8 +
> 8) bytes [ ignoring alignment for time being]
> So, the possible improvement here is, instead of creating new Long obj, we
> can keep existing inode object for tracking. Advantage is, Inode object
> already maintained in NN, so no new object creation is needed. So, we just
> need to maintain one obj ref. Above two points should significantly reduce
> the memory requirements of SPS. So, for SPS call: 8bytes for called inode
> tracking + 8 bytes for Xattr ref.
> # Use LightWeightLinkedSet instead of using LinkedList for from Q. This will
> reduce unnecessary Node creations inside LinkedList.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12951) Incorrect javadoc in SaslDataTransferServer.java#receive
[
https://issues.apache.org/jira/browse/HDFS-12951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-12951:
-
Summary: Incorrect javadoc in SaslDataTransferServer.java#receive (was:
Correct javadoc in SaslDataTransferServer.java#receive )
> Incorrect javadoc in SaslDataTransferServer.java#receive
> -
>
> Key: HDFS-12951
> URL: https://issues.apache.org/jira/browse/HDFS-12951
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-12951.001.patch
>
>
> The javadoc for the receive incorrectly states "int" in the 4th parameter.
> This should be corrected to remove the int.
> {code}
> /**
>* Receives SASL negotiation from a peer on behalf of a server.
>*
>* @param peer connection peer
>* @param underlyingOut connection output stream
>* @param underlyingIn connection input stream
>* @param int xferPort data transfer port of DataNode accepting connection
>* @param datanodeId ID of DataNode accepting connection
>* @return new pair of streams, wrapped after SASL negotiation
>* @throws IOException for any error
>*/
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12951) Correct javadoc in SaslDataTransferServer.java#receive
[
https://issues.apache.org/jira/browse/HDFS-12951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-12951:
-
Status: Patch Available (was: Open)
> Correct javadoc in SaslDataTransferServer.java#receive
> ---
>
> Key: HDFS-12951
> URL: https://issues.apache.org/jira/browse/HDFS-12951
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-12951.001.patch
>
>
> The javadoc for the receive incorrectly states "int" in the 4th parameter.
> This should be corrected to remove the int.
> {code}
> /**
>* Receives SASL negotiation from a peer on behalf of a server.
>*
>* @param peer connection peer
>* @param underlyingOut connection output stream
>* @param underlyingIn connection input stream
>* @param int xferPort data transfer port of DataNode accepting connection
>* @param datanodeId ID of DataNode accepting connection
>* @return new pair of streams, wrapped after SASL negotiation
>* @throws IOException for any error
>*/
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12951) Correct javadoc in SaslDataTransferServer.java#receive
[
https://issues.apache.org/jira/browse/HDFS-12951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-12951:
-
Attachment: HDFS-12951.001.patch
> Correct javadoc in SaslDataTransferServer.java#receive
> ---
>
> Key: HDFS-12951
> URL: https://issues.apache.org/jira/browse/HDFS-12951
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-12951.001.patch
>
>
> The javadoc for the receive incorrectly states "int" in the 4th parameter.
> This should be corrected to remove the int.
> {code}
> /**
>* Receives SASL negotiation from a peer on behalf of a server.
>*
>* @param peer connection peer
>* @param underlyingOut connection output stream
>* @param underlyingIn connection input stream
>* @param int xferPort data transfer port of DataNode accepting connection
>* @param datanodeId ID of DataNode accepting connection
>* @return new pair of streams, wrapped after SASL negotiation
>* @throws IOException for any error
>*/
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12951) Correct javadoc in SaslDataTransferServer.java#receive
Mukul Kumar Singh created HDFS-12951:
Summary: Correct javadoc in SaslDataTransferServer.java#receive
Key: HDFS-12951
URL: https://issues.apache.org/jira/browse/HDFS-12951
Project: Hadoop HDFS
Issue Type: Bug
Components: encryption
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh
The javadoc for the receive incorrectly states "int" in the 4th parameter. This
should be corrected to remove the int.
{code}
/**
* Receives SASL negotiation from a peer on behalf of a server.
*
* @param peer connection peer
* @param underlyingOut connection output stream
* @param underlyingIn connection input stream
* @param int xferPort data transfer port of DataNode accepting connection
* @param datanodeId ID of DataNode accepting connection
* @return new pair of streams, wrapped after SASL negotiation
* @throws IOException for any error
*/
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12949) Fix findbugs warning in ImageWriter.java
[
https://issues.apache.org/jira/browse/HDFS-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298618#comment-16298618
]
genericqa commented on HDFS-12949:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
26m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-fs2img in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12949 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903057/HDFS-12949.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 5b8276780edd 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d62932c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/22472/testReport/ |
| Max. process+thread count | 340 (vs. ulimit of 5000) |
| modules | C: hadoop-tools/hadoop-fs2img U: hadoop-tools/hadoop-fs2img |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/22472/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fix findbugs warning in ImageWriter.java
>
>
> Key: HDFS-12949
> URL: https://issues.apache.org/jira/browse/HDFS-12949
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
> Attachments: HDFS-12949.001.patch
>
>
> There are 1 findbugs warning in hadoop-fs2img module.
> {noformat}
> OBL new
> org.apach
[jira] [Commented] (HDFS-12950) oiv will fail in secure cluster
[
https://issues.apache.org/jira/browse/HDFS-12950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298604#comment-16298604
]
Brahma Reddy Battula commented on HDFS-12950:
-
Can be avioded by changing the configurations,but user will not be knowing
about this. So we might need to document Or pass ENV OR implement dummy
delegationtoken..?
Any thoughts.?
> oiv will fail in secure cluster
>
>
> Key: HDFS-12950
> URL: https://issues.apache.org/jira/browse/HDFS-12950
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>
> We we execute ls, it will throw following.
> {noformat}
> hdfs dfs -ls webhdfs://127.0.0.1:5978/
> ls: Invalid value for webhdfs parameter "op"
> {noformat}
> When client is configured with security (i.e "hadoop.security.authentication=
> KERBEROS) ,
> then webhdfs will request getdelegation token which is not implemented and
> hence it will throw “ls: Invalid value for webhdfs parameter "op"”.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12950) oiv will fail in secure cluster
Brahma Reddy Battula created HDFS-12950:
---
Summary: oiv will fail in secure cluster
Key: HDFS-12950
URL: https://issues.apache.org/jira/browse/HDFS-12950
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Brahma Reddy Battula
Assignee: Brahma Reddy Battula
We we execute ls, it will throw following.
{noformat}
hdfs dfs -ls webhdfs://127.0.0.1:5978/
ls: Invalid value for webhdfs parameter "op"
{noformat}
When client is configured with security (i.e "hadoop.security.authentication=
KERBEROS) ,
then webhdfs will request getdelegation token which is not implemented and
hence it will throw “ls: Invalid value for webhdfs parameter "op"”.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up
[ https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298591#comment-16298591 ] Brahma Reddy Battula commented on HDFS-12935: - [~jiangjianfei] thanks for reporting and patch.The apporach looks good to me. Added you as HDFS contributor,now onwards you can assign issues yourself.Have look at [HowToContribute|https://wiki.apache.org/hadoop/HowToContribute] Removed the fixversion which will be set whil committing the patch. Can we handle for remainging admin commands(which are not require to persist) as part of this jira apart from safemode(HDFS-8277)..? hdfs dfsadmin -saveNamespace hdfs dfsadmin -refreshNodes hdfs dfsadmin -refreshCallQueue and if any command I miss here. > Get ambiguous result for DFSAdmin command in HA mode when only one namenode > is up > - > > Key: HDFS-12935 > URL: https://issues.apache.org/jira/browse/HDFS-12935 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 3.0.0-beta1 >Reporter: Jianfei Jiang > Attachments: HDFS_12935.001.patch > > > In HA mode, if one namenode is down, most of functions can still work. When > considering the following two occasions: > (1)nn1 up and nn2 down > (2)nn1 down and nn2 up > These two occasions should be equivalent. However, some of the DFSAdmin > commands will have ambiguous results. The commands can be send successfully > to the up namenode and are always functionally useful only when nn1 is up > regardless of exception (IOException when connecting to the down namenode > nn2). If only nn2 is up, the commands have no use at all and only exception > to connect nn1 can be found. > See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to > set balancer bandwidth value for datanodes as an example. It works and all > the datanodes can get the setting values only when nn1 is up. If only nn2 is > up, the command throws exception directly and no datanode get the bandwidth > setting. Approximately ten DFSAdmin commands use the similar logical process > and may be ambiguous. > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345 > *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820* > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei02:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234 > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei01:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up
[ https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula reassigned HDFS-12935: --- Assignee: Jianfei Jiang > Get ambiguous result for DFSAdmin command in HA mode when only one namenode > is up > - > > Key: HDFS-12935 > URL: https://issues.apache.org/jira/browse/HDFS-12935 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 3.0.0-beta1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang > Attachments: HDFS_12935.001.patch > > > In HA mode, if one namenode is down, most of functions can still work. When > considering the following two occasions: > (1)nn1 up and nn2 down > (2)nn1 down and nn2 up > These two occasions should be equivalent. However, some of the DFSAdmin > commands will have ambiguous results. The commands can be send successfully > to the up namenode and are always functionally useful only when nn1 is up > regardless of exception (IOException when connecting to the down namenode > nn2). If only nn2 is up, the commands have no use at all and only exception > to connect nn1 can be found. > See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to > set balancer bandwidth value for datanodes as an example. It works and all > the datanodes can get the setting values only when nn1 is up. If only nn2 is > up, the command throws exception directly and no datanode get the bandwidth > setting. Approximately ten DFSAdmin commands use the similar logical process > and may be ambiguous. > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345 > *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820* > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei02:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234 > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei01:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up
[ https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12935: Fix Version/s: (was: 3.0.0-beta1) > Get ambiguous result for DFSAdmin command in HA mode when only one namenode > is up > - > > Key: HDFS-12935 > URL: https://issues.apache.org/jira/browse/HDFS-12935 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 3.0.0-beta1 >Reporter: Jianfei Jiang > Attachments: HDFS_12935.001.patch > > > In HA mode, if one namenode is down, most of functions can still work. When > considering the following two occasions: > (1)nn1 up and nn2 down > (2)nn1 down and nn2 up > These two occasions should be equivalent. However, some of the DFSAdmin > commands will have ambiguous results. The commands can be send successfully > to the up namenode and are always functionally useful only when nn1 is up > regardless of exception (IOException when connecting to the down namenode > nn2). If only nn2 is up, the commands have no use at all and only exception > to connect nn1 can be found. > See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to > set balancer bandwidth value for datanodes as an example. It works and all > the datanodes can get the setting values only when nn1 is up. If only nn2 is > up, the command throws exception directly and no datanode get the bandwidth > setting. Approximately ten DFSAdmin commands use the similar logical process > and may be ambiguous. > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345 > *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820* > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei02:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234 > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei01:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12941) Ozone: ConfServlet does not trim values during the description parsing
[
https://issues.apache.org/jira/browse/HDFS-12941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-12941:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-7240
Target Version/s: HDFS-7240
Status: Resolved (was: Patch Available)
Thanks for the contribution [~elek]. I have committed it to feature branch.
> Ozone: ConfServlet does not trim values during the description parsing
> --
>
> Key: HDFS-12941
> URL: https://issues.apache.org/jira/browse/HDFS-12941
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Fix For: HDFS-7240
>
> Attachments: HDFS-12941-HDFS-7240.001.patch
>
>
> In ozone-default.xml an unnecessary space is added to the
> ozone.open.key.expire.threshold key name.
> It causes a NPE in ConfServlet:
> {code}
> Server ErrorCaused
> by:java.lang.NullPointerException
> at
> org.apache.hadoop.conf.ConfServlet.lambda$processConfigTagRequest$0(ConfServlet.java:147)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at
> java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
> at
> org.apache.hadoop.conf.ConfServlet.processConfigTagRequest(ConfServlet.java:143)
> at org.apache.hadoop.conf.ConfServlet.doGet(ConfServlet.java:101)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:110)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1578)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> The problem is here:
> {code}
> Properties properties = config.getAllPropertiesByTags(tagList);
> if (propertyMap == null) {
> loadDescriptions();
> }
> List filteredProp
[jira] [Commented] (HDFS-12949) Fix findbugs warning in ImageWriter.java
[
https://issues.apache.org/jira/browse/HDFS-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298550#comment-16298550
]
Akira Ajisaka commented on HDFS-12949:
--
Hi [~chris.douglas], would you review this?
> Fix findbugs warning in ImageWriter.java
>
>
> Key: HDFS-12949
> URL: https://issues.apache.org/jira/browse/HDFS-12949
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
> Attachments: HDFS-12949.001.patch
>
>
> There are 1 findbugs warning in hadoop-fs2img module.
> {noformat}
> OBL new
> org.apache.hadoop.hdfs.server.namenode.ImageWriter(ImageWriter$Options) may
> fail to clean up java.io.OutputStream on checked exception
> Bug type OBL_UNSATISFIED_OBLIGATION_EXCEPTION_EDGE (click for details)
> In class org.apache.hadoop.hdfs.server.namenode.ImageWriter
> In method new
> org.apache.hadoop.hdfs.server.namenode.ImageWriter(ImageWriter$Options)
> Reference type java.io.OutputStream
> 1 instances of obligation remaining
> Obligation to clean up resource created at ImageWriter.java:[line 184] is not
> discharged
> Path continues at ImageWriter.java:[line 185]
> Path continues at ImageWriter.java:[line 186]
> Path continues at ImageWriter.java:[line 187]
> Path continues at ImageWriter.java:[line 188]
> Remaining obligations: {OutputStream x 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12949) Fix findbugs warning in ImageWriter.java
[
https://issues.apache.org/jira/browse/HDFS-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-12949:
-
Status: Patch Available (was: Open)
> Fix findbugs warning in ImageWriter.java
>
>
> Key: HDFS-12949
> URL: https://issues.apache.org/jira/browse/HDFS-12949
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
> Attachments: HDFS-12949.001.patch
>
>
> There are 1 findbugs warning in hadoop-fs2img module.
> {noformat}
> OBL new
> org.apache.hadoop.hdfs.server.namenode.ImageWriter(ImageWriter$Options) may
> fail to clean up java.io.OutputStream on checked exception
> Bug type OBL_UNSATISFIED_OBLIGATION_EXCEPTION_EDGE (click for details)
> In class org.apache.hadoop.hdfs.server.namenode.ImageWriter
> In method new
> org.apache.hadoop.hdfs.server.namenode.ImageWriter(ImageWriter$Options)
> Reference type java.io.OutputStream
> 1 instances of obligation remaining
> Obligation to clean up resource created at ImageWriter.java:[line 184] is not
> discharged
> Path continues at ImageWriter.java:[line 185]
> Path continues at ImageWriter.java:[line 186]
> Path continues at ImageWriter.java:[line 187]
> Path continues at ImageWriter.java:[line 188]
> Remaining obligations: {OutputStream x 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12949) Fix findbugs warning in ImageWriter.java
[
https://issues.apache.org/jira/browse/HDFS-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-12949:
-
Attachment: HDFS-12949.001.patch
HDFS-12903 added findbugs-exclude.xml to ignore the warning, however, it is not
sufficient. We need to add a setting in pom.xml to use the findbugs-exclude.xml.
> Fix findbugs warning in ImageWriter.java
>
>
> Key: HDFS-12949
> URL: https://issues.apache.org/jira/browse/HDFS-12949
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
> Attachments: HDFS-12949.001.patch
>
>
> There are 1 findbugs warning in hadoop-fs2img module.
> {noformat}
> OBL new
> org.apache.hadoop.hdfs.server.namenode.ImageWriter(ImageWriter$Options) may
> fail to clean up java.io.OutputStream on checked exception
> Bug type OBL_UNSATISFIED_OBLIGATION_EXCEPTION_EDGE (click for details)
> In class org.apache.hadoop.hdfs.server.namenode.ImageWriter
> In method new
> org.apache.hadoop.hdfs.server.namenode.ImageWriter(ImageWriter$Options)
> Reference type java.io.OutputStream
> 1 instances of obligation remaining
> Obligation to clean up resource created at ImageWriter.java:[line 184] is not
> discharged
> Path continues at ImageWriter.java:[line 185]
> Path continues at ImageWriter.java:[line 186]
> Path continues at ImageWriter.java:[line 187]
> Path continues at ImageWriter.java:[line 188]
> Remaining obligations: {OutputStream x 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
