[jira] [Commented] (HDFS-13246) FileInputStream redundant closes in readReplicasFromCache
[
https://issues.apache.org/jira/browse/HDFS-13246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399946#comment-16399946
]
liaoyuxiangqin commented on HDFS-13246:
---
[~xiaochen] Thanks for your give some suggestions and committed, and
[~shahrs87] for review.
> FileInputStream redundant closes in readReplicasFromCache
> --
>
> Key: HDFS-13246
> URL: https://issues.apache.org/jira/browse/HDFS-13246
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: liaoyuxiangqin
>Assignee: liaoyuxiangqin
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HDFS-13246.001.patch
>
>
> When i read the readReplicasFromCache() of BlockPoolSlice class in datanode,
> I found the following code closes fileinputstream redundant, I think
> IOUtils.closeStream(inputStream) in finally code block could guarantee close
> the inputStream correctly, So the
> inputStream.close() can remove. Thanks.
>
> {code:java|title=BlockPoolSlice.java|borderStyle=solid}
> FileInputStream inputStream = null;
> try {
> inputStream = fileIoProvider.getFileInputStream(volume, replicaFile);
> BlockListAsLongs blocksList =
> BlockListAsLongs.readFrom(inputStream, maxDataLength);
> if (blocksList == null) {
> return false;
> }
> for (BlockReportReplica replica : blocksList) {
> switch (replica.getState()) {
> case FINALIZED:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, true);
> break;
> case RUR:
> case RBW:
> case RWR:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, false);
> break;
> default:
> break;
> }
> }
> inputStream.close();
> // Now it is safe to add the replica into volumeMap
> // In case of any exception during parsing this cache file, fall back
> // to scan all the files on disk.
> for (Iterator iter =
> tmpReplicaMap.replicas(bpid).iterator(); iter.hasNext(); ) {
> ReplicaInfo info = iter.next();
> // We use a lightweight GSet to store replicaInfo, we need to remove
> // it from one GSet before adding to another.
> iter.remove();
> volumeMap.add(bpid, info);
> }
> LOG.info("Successfully read replica from cache file : "
> + replicaFile.getPath());
> return true;
> } catch (Exception e) {
> // Any exception we need to revert back to read from disk
> // Log the error and return false
> LOG.info("Exception occurred while reading the replicas cache file: "
> + replicaFile.getPath(), e );
> return false;
> }
> finally {
> if (!fileIoProvider.delete(volume, replicaFile)) {
> LOG.info("Failed to delete replica cache file: " +
> replicaFile.getPath());
> }
> // close the inputStream
> IOUtils.closeStream(inputStream);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13163) Move invalidated blocks to replica-trash with disk layout based on timestamp
[ https://issues.apache.org/jira/browse/HDFS-13163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399941#comment-16399941 ] Bharat Viswanadham commented on HDFS-13163: --- Attached patch v04 to fix asflicenese errors. Ran tests locally, and failed test cases passed. > Move invalidated blocks to replica-trash with disk layout based on timestamp > > > Key: HDFS-13163 > URL: https://issues.apache.org/jira/browse/HDFS-13163 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13163-HDFS-12996.00.patch, > HDFS-13163-HDFS-12996.01.patch, HDFS-13163-HDFS-12996.02.patch, > HDFS-13163-HDFS-12996.03.patch, HDFS-13163-HDFS-12996.04.patch > > > When Blocks are invalidated, move the blocks to replica-trash directory and > place it in the folder when the invalidate is received from the namenode > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13163) Move invalidated blocks to replica-trash with disk layout based on timestamp
[ https://issues.apache.org/jira/browse/HDFS-13163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-13163: -- Attachment: HDFS-13163-HDFS-12996.04.patch > Move invalidated blocks to replica-trash with disk layout based on timestamp > > > Key: HDFS-13163 > URL: https://issues.apache.org/jira/browse/HDFS-13163 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13163-HDFS-12996.00.patch, > HDFS-13163-HDFS-12996.01.patch, HDFS-13163-HDFS-12996.02.patch, > HDFS-13163-HDFS-12996.03.patch, HDFS-13163-HDFS-12996.04.patch > > > When Blocks are invalidated, move the blocks to replica-trash directory and > place it in the folder when the invalidate is received from the namenode > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13246) FileInputStream redundant closes in readReplicasFromCache
[
https://issues.apache.org/jira/browse/HDFS-13246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399933#comment-16399933
]
Hudson commented on HDFS-13246:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13842 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13842/])
HDFS-13246. FileInputStream redundant closes in readReplicasFromCache. (xiao:
rev 5ff22d4c3a417292c496a4bb02e8b19fad66ec63)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
> FileInputStream redundant closes in readReplicasFromCache
> --
>
> Key: HDFS-13246
> URL: https://issues.apache.org/jira/browse/HDFS-13246
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: liaoyuxiangqin
>Assignee: liaoyuxiangqin
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HDFS-13246.001.patch
>
>
> When i read the readReplicasFromCache() of BlockPoolSlice class in datanode,
> I found the following code closes fileinputstream redundant, I think
> IOUtils.closeStream(inputStream) in finally code block could guarantee close
> the inputStream correctly, So the
> inputStream.close() can remove. Thanks.
>
> {code:java|title=BlockPoolSlice.java|borderStyle=solid}
> FileInputStream inputStream = null;
> try {
> inputStream = fileIoProvider.getFileInputStream(volume, replicaFile);
> BlockListAsLongs blocksList =
> BlockListAsLongs.readFrom(inputStream, maxDataLength);
> if (blocksList == null) {
> return false;
> }
> for (BlockReportReplica replica : blocksList) {
> switch (replica.getState()) {
> case FINALIZED:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, true);
> break;
> case RUR:
> case RBW:
> case RWR:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, false);
> break;
> default:
> break;
> }
> }
> inputStream.close();
> // Now it is safe to add the replica into volumeMap
> // In case of any exception during parsing this cache file, fall back
> // to scan all the files on disk.
> for (Iterator iter =
> tmpReplicaMap.replicas(bpid).iterator(); iter.hasNext(); ) {
> ReplicaInfo info = iter.next();
> // We use a lightweight GSet to store replicaInfo, we need to remove
> // it from one GSet before adding to another.
> iter.remove();
> volumeMap.add(bpid, info);
> }
> LOG.info("Successfully read replica from cache file : "
> + replicaFile.getPath());
> return true;
> } catch (Exception e) {
> // Any exception we need to revert back to read from disk
> // Log the error and return false
> LOG.info("Exception occurred while reading the replicas cache file: "
> + replicaFile.getPath(), e );
> return false;
> }
> finally {
> if (!fileIoProvider.delete(volume, replicaFile)) {
> LOG.info("Failed to delete replica cache file: " +
> replicaFile.getPath());
> }
> // close the inputStream
> IOUtils.closeStream(inputStream);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13239) Fix non-empty dir warning message when setting default EC policy
[
https://issues.apache.org/jira/browse/HDFS-13239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399925#comment-16399925
]
Bharat Viswanadham commented on HDFS-13239:
---
Thank You [~xiaochen] and [~hanishakoneru] for review and committing the patch.
> Fix non-empty dir warning message when setting default EC policy
>
>
> Key: HDFS-13239
> URL: https://issues.apache.org/jira/browse/HDFS-13239
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Hanisha Koneru
>Assignee: Bharat Viswanadham
>Priority: Minor
> Fix For: 3.1.0, 3.0.2
>
> Attachments: HDFS-13239.00.patch, HDFS-13239.01.patch,
> HDFS-13239.02.patch, HDFS-13239.03.patch, HDFS-13239.04.patch
>
>
> When EC policy is set on a non-empty directory, the following warning message
> is given:
> {code}
> $hdfs ec -setPolicy -policy RS-6-3-1024k -path /ec1
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to RS-6-3-1024k
> {code}
> When we do not specify the -policy parameter when setting EC policy on a
> directory, it takes the default EC policy. Setting default EC policy in this
> way on a non-empty directory gives the following warning message:
> {code}
> $hdfs ec -setPolicy -path /ec2
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to null
> {code}
> Notice that the warning message in the 2nd case has the ecPolicy name shown
> as null. We should instead give the default EC policy name in this message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13235) DiskBalancer: Update Documentation to add newly added options
[ https://issues.apache.org/jira/browse/HDFS-13235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399923#comment-16399923 ] Bharat Viswanadham commented on HDFS-13235: --- Thank You [~arpitagarwal] for review and committing the patch. > DiskBalancer: Update Documentation to add newly added options > - > > Key: HDFS-13235 > URL: https://issues.apache.org/jira/browse/HDFS-13235 > Project: Hadoop HDFS > Issue Type: Improvement > Components: diskbalancer, documentation >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Fix For: 3.2.0 > > Attachments: HDFS-13235.00.patch, HDFS-13235.01.patch > > > HDFS-13181 added dfs.disk.balancer.plan.valid.interval > HDFS-13178 added skipDateCheck Option -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-336) dfsadmin -report should report number of blocks from datanode
[ https://issues.apache.org/jira/browse/HDFS-336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399922#comment-16399922 ] Bharat Viswanadham commented on HDFS-336: - Thank You [~arpitagarwal] for review and committing the patch. > dfsadmin -report should report number of blocks from datanode > - > > Key: HDFS-336 > URL: https://issues.apache.org/jira/browse/HDFS-336 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Lohit Vijayarenu >Assignee: Bharat Viswanadham >Priority: Minor > Labels: newbie > Fix For: 3.2.0 > > Attachments: HDFS-336.00.patch, HDFS-336.01.patch, HDFS-336.02.patch > > > _hadoop dfsadmin -report_ seems to miss number of blocks from a datanode. > Number of blocks hosted by a datanode is a good info which should be included > in the report. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13246) FileInputStream redundant closes in readReplicasFromCache
[
https://issues.apache.org/jira/browse/HDFS-13246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399913#comment-16399913
]
Xiao Chen edited comment on HDFS-13246 at 3/15/18 5:04 AM:
---
Failed tests look unrelated
Committed to trunk. Thanks for the contribution [~liaoyuxiangqin], and Rushabh
for review
was (Author: xiaochen):
Committed to trunk. Thanks for the contribution [~liaoyuxiangqin], and Rushabh
for review
> FileInputStream redundant closes in readReplicasFromCache
> --
>
> Key: HDFS-13246
> URL: https://issues.apache.org/jira/browse/HDFS-13246
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: liaoyuxiangqin
>Assignee: liaoyuxiangqin
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HDFS-13246.001.patch
>
>
> When i read the readReplicasFromCache() of BlockPoolSlice class in datanode,
> I found the following code closes fileinputstream redundant, I think
> IOUtils.closeStream(inputStream) in finally code block could guarantee close
> the inputStream correctly, So the
> inputStream.close() can remove. Thanks.
>
> {code:java|title=BlockPoolSlice.java|borderStyle=solid}
> FileInputStream inputStream = null;
> try {
> inputStream = fileIoProvider.getFileInputStream(volume, replicaFile);
> BlockListAsLongs blocksList =
> BlockListAsLongs.readFrom(inputStream, maxDataLength);
> if (blocksList == null) {
> return false;
> }
> for (BlockReportReplica replica : blocksList) {
> switch (replica.getState()) {
> case FINALIZED:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, true);
> break;
> case RUR:
> case RBW:
> case RWR:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, false);
> break;
> default:
> break;
> }
> }
> inputStream.close();
> // Now it is safe to add the replica into volumeMap
> // In case of any exception during parsing this cache file, fall back
> // to scan all the files on disk.
> for (Iterator iter =
> tmpReplicaMap.replicas(bpid).iterator(); iter.hasNext(); ) {
> ReplicaInfo info = iter.next();
> // We use a lightweight GSet to store replicaInfo, we need to remove
> // it from one GSet before adding to another.
> iter.remove();
> volumeMap.add(bpid, info);
> }
> LOG.info("Successfully read replica from cache file : "
> + replicaFile.getPath());
> return true;
> } catch (Exception e) {
> // Any exception we need to revert back to read from disk
> // Log the error and return false
> LOG.info("Exception occurred while reading the replicas cache file: "
> + replicaFile.getPath(), e );
> return false;
> }
> finally {
> if (!fileIoProvider.delete(volume, replicaFile)) {
> LOG.info("Failed to delete replica cache file: " +
> replicaFile.getPath());
> }
> // close the inputStream
> IOUtils.closeStream(inputStream);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13246) FileInputStream redundant closes in readReplicasFromCache
[
https://issues.apache.org/jira/browse/HDFS-13246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-13246:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.2.0
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks for the contribution [~liaoyuxiangqin], and Rushabh
for review
> FileInputStream redundant closes in readReplicasFromCache
> --
>
> Key: HDFS-13246
> URL: https://issues.apache.org/jira/browse/HDFS-13246
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: liaoyuxiangqin
>Assignee: liaoyuxiangqin
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HDFS-13246.001.patch
>
>
> When i read the readReplicasFromCache() of BlockPoolSlice class in datanode,
> I found the following code closes fileinputstream redundant, I think
> IOUtils.closeStream(inputStream) in finally code block could guarantee close
> the inputStream correctly, So the
> inputStream.close() can remove. Thanks.
>
> {code:java|title=BlockPoolSlice.java|borderStyle=solid}
> FileInputStream inputStream = null;
> try {
> inputStream = fileIoProvider.getFileInputStream(volume, replicaFile);
> BlockListAsLongs blocksList =
> BlockListAsLongs.readFrom(inputStream, maxDataLength);
> if (blocksList == null) {
> return false;
> }
> for (BlockReportReplica replica : blocksList) {
> switch (replica.getState()) {
> case FINALIZED:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, true);
> break;
> case RUR:
> case RBW:
> case RWR:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, false);
> break;
> default:
> break;
> }
> }
> inputStream.close();
> // Now it is safe to add the replica into volumeMap
> // In case of any exception during parsing this cache file, fall back
> // to scan all the files on disk.
> for (Iterator iter =
> tmpReplicaMap.replicas(bpid).iterator(); iter.hasNext(); ) {
> ReplicaInfo info = iter.next();
> // We use a lightweight GSet to store replicaInfo, we need to remove
> // it from one GSet before adding to another.
> iter.remove();
> volumeMap.add(bpid, info);
> }
> LOG.info("Successfully read replica from cache file : "
> + replicaFile.getPath());
> return true;
> } catch (Exception e) {
> // Any exception we need to revert back to read from disk
> // Log the error and return false
> LOG.info("Exception occurred while reading the replicas cache file: "
> + replicaFile.getPath(), e );
> return false;
> }
> finally {
> if (!fileIoProvider.delete(volume, replicaFile)) {
> LOG.info("Failed to delete replica cache file: " +
> replicaFile.getPath());
> }
> // close the inputStream
> IOUtils.closeStream(inputStream);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13246) FileInputStream redundant closes in readReplicasFromCache
[
https://issues.apache.org/jira/browse/HDFS-13246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-13246:
-
Priority: Minor (was: Major)
> FileInputStream redundant closes in readReplicasFromCache
> --
>
> Key: HDFS-13246
> URL: https://issues.apache.org/jira/browse/HDFS-13246
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: liaoyuxiangqin
>Assignee: liaoyuxiangqin
>Priority: Minor
> Attachments: HDFS-13246.001.patch
>
>
> When i read the readReplicasFromCache() of BlockPoolSlice class in datanode,
> I found the following code closes fileinputstream redundant, I think
> IOUtils.closeStream(inputStream) in finally code block could guarantee close
> the inputStream correctly, So the
> inputStream.close() can remove. Thanks.
>
> {code:java|title=BlockPoolSlice.java|borderStyle=solid}
> FileInputStream inputStream = null;
> try {
> inputStream = fileIoProvider.getFileInputStream(volume, replicaFile);
> BlockListAsLongs blocksList =
> BlockListAsLongs.readFrom(inputStream, maxDataLength);
> if (blocksList == null) {
> return false;
> }
> for (BlockReportReplica replica : blocksList) {
> switch (replica.getState()) {
> case FINALIZED:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, true);
> break;
> case RUR:
> case RBW:
> case RWR:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, false);
> break;
> default:
> break;
> }
> }
> inputStream.close();
> // Now it is safe to add the replica into volumeMap
> // In case of any exception during parsing this cache file, fall back
> // to scan all the files on disk.
> for (Iterator iter =
> tmpReplicaMap.replicas(bpid).iterator(); iter.hasNext(); ) {
> ReplicaInfo info = iter.next();
> // We use a lightweight GSet to store replicaInfo, we need to remove
> // it from one GSet before adding to another.
> iter.remove();
> volumeMap.add(bpid, info);
> }
> LOG.info("Successfully read replica from cache file : "
> + replicaFile.getPath());
> return true;
> } catch (Exception e) {
> // Any exception we need to revert back to read from disk
> // Log the error and return false
> LOG.info("Exception occurred while reading the replicas cache file: "
> + replicaFile.getPath(), e );
> return false;
> }
> finally {
> if (!fileIoProvider.delete(volume, replicaFile)) {
> LOG.info("Failed to delete replica cache file: " +
> replicaFile.getPath());
> }
> // close the inputStream
> IOUtils.closeStream(inputStream);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13246) FileInputStream redundant closes in readReplicasFromCache
[
https://issues.apache.org/jira/browse/HDFS-13246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399912#comment-16399912
]
Xiao Chen commented on HDFS-13246:
--
Trunk can use try-with-resources to make it slightly simpler. But I'll commit
as-is in case people wanted to backport to branch-2 in the future. +1
> FileInputStream redundant closes in readReplicasFromCache
> --
>
> Key: HDFS-13246
> URL: https://issues.apache.org/jira/browse/HDFS-13246
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: liaoyuxiangqin
>Assignee: liaoyuxiangqin
>Priority: Major
> Attachments: HDFS-13246.001.patch
>
>
> When i read the readReplicasFromCache() of BlockPoolSlice class in datanode,
> I found the following code closes fileinputstream redundant, I think
> IOUtils.closeStream(inputStream) in finally code block could guarantee close
> the inputStream correctly, So the
> inputStream.close() can remove. Thanks.
>
> {code:java|title=BlockPoolSlice.java|borderStyle=solid}
> FileInputStream inputStream = null;
> try {
> inputStream = fileIoProvider.getFileInputStream(volume, replicaFile);
> BlockListAsLongs blocksList =
> BlockListAsLongs.readFrom(inputStream, maxDataLength);
> if (blocksList == null) {
> return false;
> }
> for (BlockReportReplica replica : blocksList) {
> switch (replica.getState()) {
> case FINALIZED:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, true);
> break;
> case RUR:
> case RBW:
> case RWR:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, false);
> break;
> default:
> break;
> }
> }
> inputStream.close();
> // Now it is safe to add the replica into volumeMap
> // In case of any exception during parsing this cache file, fall back
> // to scan all the files on disk.
> for (Iterator iter =
> tmpReplicaMap.replicas(bpid).iterator(); iter.hasNext(); ) {
> ReplicaInfo info = iter.next();
> // We use a lightweight GSet to store replicaInfo, we need to remove
> // it from one GSet before adding to another.
> iter.remove();
> volumeMap.add(bpid, info);
> }
> LOG.info("Successfully read replica from cache file : "
> + replicaFile.getPath());
> return true;
> } catch (Exception e) {
> // Any exception we need to revert back to read from disk
> // Log the error and return false
> LOG.info("Exception occurred while reading the replicas cache file: "
> + replicaFile.getPath(), e );
> return false;
> }
> finally {
> if (!fileIoProvider.delete(volume, replicaFile)) {
> LOG.info("Failed to delete replica cache file: " +
> replicaFile.getPath());
> }
> // close the inputStream
> IOUtils.closeStream(inputStream);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13246) FileInputStream redundant closes in readReplicasFromCache
[
https://issues.apache.org/jira/browse/HDFS-13246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen reassigned HDFS-13246:
Assignee: liaoyuxiangqin
> FileInputStream redundant closes in readReplicasFromCache
> --
>
> Key: HDFS-13246
> URL: https://issues.apache.org/jira/browse/HDFS-13246
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: liaoyuxiangqin
>Assignee: liaoyuxiangqin
>Priority: Major
> Attachments: HDFS-13246.001.patch
>
>
> When i read the readReplicasFromCache() of BlockPoolSlice class in datanode,
> I found the following code closes fileinputstream redundant, I think
> IOUtils.closeStream(inputStream) in finally code block could guarantee close
> the inputStream correctly, So the
> inputStream.close() can remove. Thanks.
>
> {code:java|title=BlockPoolSlice.java|borderStyle=solid}
> FileInputStream inputStream = null;
> try {
> inputStream = fileIoProvider.getFileInputStream(volume, replicaFile);
> BlockListAsLongs blocksList =
> BlockListAsLongs.readFrom(inputStream, maxDataLength);
> if (blocksList == null) {
> return false;
> }
> for (BlockReportReplica replica : blocksList) {
> switch (replica.getState()) {
> case FINALIZED:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, true);
> break;
> case RUR:
> case RBW:
> case RWR:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, false);
> break;
> default:
> break;
> }
> }
> inputStream.close();
> // Now it is safe to add the replica into volumeMap
> // In case of any exception during parsing this cache file, fall back
> // to scan all the files on disk.
> for (Iterator iter =
> tmpReplicaMap.replicas(bpid).iterator(); iter.hasNext(); ) {
> ReplicaInfo info = iter.next();
> // We use a lightweight GSet to store replicaInfo, we need to remove
> // it from one GSet before adding to another.
> iter.remove();
> volumeMap.add(bpid, info);
> }
> LOG.info("Successfully read replica from cache file : "
> + replicaFile.getPath());
> return true;
> } catch (Exception e) {
> // Any exception we need to revert back to read from disk
> // Log the error and return false
> LOG.info("Exception occurred while reading the replicas cache file: "
> + replicaFile.getPath(), e );
> return false;
> }
> finally {
> if (!fileIoProvider.delete(volume, replicaFile)) {
> LOG.info("Failed to delete replica cache file: " +
> replicaFile.getPath());
> }
> // close the inputStream
> IOUtils.closeStream(inputStream);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-13288) Why we don't add a harder lease expiration limit.
[ https://issues.apache.org/jira/browse/HDFS-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igloo resolved HDFS-13288. -- Resolution: Invalid > Why we don't add a harder lease expiration limit. > - > > Key: HDFS-13288 > URL: https://issues.apache.org/jira/browse/HDFS-13288 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.6.5 >Reporter: Igloo >Priority: Minor > > Currently there exists a soft expire timeout(1 minutes by default) and hard > expire timeout(60 minutes by default). > On our production environment. Some client began writing a file long > time(more than one year) ago, when writing finished and tried to close the > output stream, the client failed closing it (for some IOException. etc. ). > But the client process is a background service, it doesn't exit. So the lease > doesn't released for more than one year. > The problem is that, the lease for the file is occupied, we have to call > recover lease on the file when doing demission or appending operation. > > So I am wondering why we don't add a more harder lease expire timeout, when a > lease lasts too long (maybe one month), revoke it. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13288) Why we don't add a harder lease expiration limit.
[ https://issues.apache.org/jira/browse/HDFS-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399878#comment-16399878 ] Igloo commented on HDFS-13288: -- [~vinayrpet] Got your point.. "Namenode renews the lease for whole client, not per file." you are right. thx~ > Why we don't add a harder lease expiration limit. > - > > Key: HDFS-13288 > URL: https://issues.apache.org/jira/browse/HDFS-13288 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.6.5 >Reporter: Igloo >Priority: Minor > > Currently there exists a soft expire timeout(1 minutes by default) and hard > expire timeout(60 minutes by default). > On our production environment. Some client began writing a file long > time(more than one year) ago, when writing finished and tried to close the > output stream, the client failed closing it (for some IOException. etc. ). > But the client process is a background service, it doesn't exit. So the lease > doesn't released for more than one year. > The problem is that, the lease for the file is occupied, we have to call > recover lease on the file when doing demission or appending operation. > > So I am wondering why we don't add a more harder lease expire timeout, when a > lease lasts too long (maybe one month), revoke it. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13251) Avoid using hard coded datanode data dirs in unit tests
[
https://issues.apache.org/jira/browse/HDFS-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399866#comment-16399866
]
Ajay Kumar commented on HDFS-13251:
---
[~xyao], thanks for commit. Added addendum patch with below 2 changes:
* TestDataNodeVolumeFailureToleration: line 162 changed getStorageDir to
getInstanceStorageDir
* TestDFSAdmin: formatting
> Avoid using hard coded datanode data dirs in unit tests
> ---
>
> Key: HDFS-13251
> URL: https://issues.apache.org/jira/browse/HDFS-13251
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Xiaoyu Yao
>Assignee: Ajay Kumar
>Priority: Major
> Fix For: 3.1.0, 3.2.0
>
> Attachments: HDFS-13251.000.patch, HDFS-13251.001.patch,
> HDFS-13251.002.patch, HDFS-13251.003.addendum.patch, HDFS-13251.003.patch
>
>
> There are a few unit tests that rely on hard-coded MiniDFSCluster data dir
> names.
>
> * TestDataNodeVolumeFailureToleration
> * TestDataNodeVolumeFailureReporting
> * TestDiskBalancerCommand
> * TestBlockStatsMXBean
> * TestDataNodeVolumeMetrics
> * TestDFSAdmin
> * TestDataNodeHotSwapVolumes
> * TestDataNodeVolumeFailure
> This ticket is opened to use
> {code:java}
> MiniDFSCluster#getInstanceStorageDir(0, 1);
> instead of like below
> new File(cluster.getDataDirectory(), "data1");{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13251) Avoid using hard coded datanode data dirs in unit tests
[
https://issues.apache.org/jira/browse/HDFS-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HDFS-13251:
--
Attachment: HDFS-13251.003.addendum.patch
> Avoid using hard coded datanode data dirs in unit tests
> ---
>
> Key: HDFS-13251
> URL: https://issues.apache.org/jira/browse/HDFS-13251
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Xiaoyu Yao
>Assignee: Ajay Kumar
>Priority: Major
> Fix For: 3.1.0, 3.2.0
>
> Attachments: HDFS-13251.000.patch, HDFS-13251.001.patch,
> HDFS-13251.002.patch, HDFS-13251.003.addendum.patch, HDFS-13251.003.patch
>
>
> There are a few unit tests that rely on hard-coded MiniDFSCluster data dir
> names.
>
> * TestDataNodeVolumeFailureToleration
> * TestDataNodeVolumeFailureReporting
> * TestDiskBalancerCommand
> * TestBlockStatsMXBean
> * TestDataNodeVolumeMetrics
> * TestDFSAdmin
> * TestDataNodeHotSwapVolumes
> * TestDataNodeVolumeFailure
> This ticket is opened to use
> {code:java}
> MiniDFSCluster#getInstanceStorageDir(0, 1);
> instead of like below
> new File(cluster.getDataDirectory(), "data1");{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13261) Fix incorrect null value check
[
https://issues.apache.org/jira/browse/HDFS-13261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399864#comment-16399864
]
Hudson commented on HDFS-13261:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13841 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13841/])
HDFS-13261. Fix incorrect null value check. Contributed by Jianfei (yqlin: rev
3a0f4bc0d513f9ffb4cf571c1ac85395e116fdcc)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/AdminHelper.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ZoneReencryptionStatus.java
> Fix incorrect null value check
> --
>
> Key: HDFS-13261
> URL: https://issues.apache.org/jira/browse/HDFS-13261
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.3, 3.0.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Minor
> Fix For: 3.1.0, 2.10.0, 3.2.0
>
> Attachments: HDFS-13261.001.patch, HDFS-13261.002-branch-2.patch,
> HDFS-13261.002.patch
>
>
>
> {code:java}
> public void setZoneName(final String name) {
> Preconditions.checkNotNull(name == null)---error parameter for
> checkNotNull
> zoneName = name;
> }
> {code}
> Like above, in 3.x, the checkNotNull function gets wrong parameter in two
> cases: \{{ZoneReencryptionStatus#setZoneName}} and
> \{{AdminHelper#HelpCommand#HelpCommand}}
> In 2.x, the error occurs once in \{{AdminHelper#HelpCommand#HelpCommand}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13288) Why we don't add a harder lease expiration limit.
[ https://issues.apache.org/jira/browse/HDFS-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399863#comment-16399863 ] Vinayakumar B commented on HDFS-13288: -- I missed the point "But the client process is a background service, it doesn't exit" before posting my above comment. Namenode renews the lease for whole client, not per file. If the client is alive, there is no way for namenode to know that it has failed to close the file. Namenode thinks that file is still being written, so it will never attempt to recover, even if "harder" limit is applied. > Why we don't add a harder lease expiration limit. > - > > Key: HDFS-13288 > URL: https://issues.apache.org/jira/browse/HDFS-13288 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.6.5 >Reporter: Igloo >Priority: Minor > > Currently there exists a soft expire timeout(1 minutes by default) and hard > expire timeout(60 minutes by default). > On our production environment. Some client began writing a file long > time(more than one year) ago, when writing finished and tried to close the > output stream, the client failed closing it (for some IOException. etc. ). > But the client process is a background service, it doesn't exit. So the lease > doesn't released for more than one year. > The problem is that, the lease for the file is occupied, we have to call > recover lease on the file when doing demission or appending operation. > > So I am wondering why we don't add a more harder lease expire timeout, when a > lease lasts too long (maybe one month), revoke it. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13261) Fix incorrect null value check
[
https://issues.apache.org/jira/browse/HDFS-13261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-13261:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.2.0
2.10.0
3.1.0
Status: Resolved (was: Patch Available)
Committed to trunk, branch-3.1 and branch-2.
Thanks [~jiangjianfei] for the contrition and thanks [~xyao] for the review.
> Fix incorrect null value check
> --
>
> Key: HDFS-13261
> URL: https://issues.apache.org/jira/browse/HDFS-13261
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.3, 3.0.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Minor
> Fix For: 3.1.0, 2.10.0, 3.2.0
>
> Attachments: HDFS-13261.001.patch, HDFS-13261.002-branch-2.patch,
> HDFS-13261.002.patch
>
>
>
> {code:java}
> public void setZoneName(final String name) {
> Preconditions.checkNotNull(name == null)---error parameter for
> checkNotNull
> zoneName = name;
> }
> {code}
> Like above, in 3.x, the checkNotNull function gets wrong parameter in two
> cases: \{{ZoneReencryptionStatus#setZoneName}} and
> \{{AdminHelper#HelpCommand#HelpCommand}}
> In 2.x, the error occurs once in \{{AdminHelper#HelpCommand#HelpCommand}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13289) TestConnectionManager#testCleanup() test case need correction
[
https://issues.apache.org/jira/browse/HDFS-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399849#comment-16399849
]
Dibyendu Karmakar commented on HDFS-13289:
--
I have the patch ready. Please somebody assign this to me.
> TestConnectionManager#testCleanup() test case need correction
> -
>
> Key: HDFS-13289
> URL: https://issues.apache.org/jira/browse/HDFS-13289
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Dibyendu Karmakar
>Priority: Minor
>
> In TestConnectionManager#testCleanup()
>
> {code:java}
> // Make sure the number of connections doesn't go below minSize
> ConnectionPool pool3 = new ConnectionPool(
> conf, TEST_NN_ADDRESS, TEST_USER3, 2, 10);
> addConnectionsToPool(pool3, 10, 0);
> poolMap.put(new ConnectionPoolId(TEST_USER2, TEST_NN_ADDRESS), pool3);
> connManager.cleanup(pool3);
> checkPoolConnections(TEST_USER3, 2, 0);
> {code}
> this part need correction
> Here new ConnectionPoolId is created with TEST_USER2 but checkPoolConnections
> is done using TEST_USER3.
> In checkPoolConnections method
> {code:java}
> if (e.getKey().getUgi() == ugi)
> {code}
> then only it will validate numOfConns and numOfActiveConns. In this case for
> TEST_USER3 ' *if* 'condition is returning *false* and if you pass any value
> to the checkPoolConnections method, the test case will pass.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13289) TestConnectionManager#testCleanup() test case need correction
[
https://issues.apache.org/jira/browse/HDFS-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dibyendu Karmakar updated HDFS-13289:
-
Description:
In TestConnectionManager#testCleanup()
{code:java}
// Make sure the number of connections doesn't go below minSize
ConnectionPool pool3 = new ConnectionPool(
conf, TEST_NN_ADDRESS, TEST_USER3, 2, 10);
addConnectionsToPool(pool3, 10, 0);
poolMap.put(new ConnectionPoolId(TEST_USER2, TEST_NN_ADDRESS), pool3);
connManager.cleanup(pool3);
checkPoolConnections(TEST_USER3, 2, 0);
{code}
this part need correction
Here new ConnectionPoolId is created with TEST_USER2 but checkPoolConnections
is done using TEST_USER3.
In checkPoolConnections method
{code:java}
if (e.getKey().getUgi() == ugi)
{code}
then only it will validate numOfConns and numOfActiveConns. In this case for
TEST_USER3 ' *if* 'condition is returning *false* and if you pass any value
to the checkPoolConnections method, the test case will pass.
was:
In TestConnectionManager#testCleanup()
{code:java}
// Make sure the number of connections doesn't go below minSize
ConnectionPool pool3 = new ConnectionPool(
conf, TEST_NN_ADDRESS, TEST_USER3, 2, 10);
addConnectionsToPool(pool3, 10, 0);
poolMap.put(new ConnectionPoolId(TEST_USER2, TEST_NN_ADDRESS), pool3);
connManager.cleanup(pool3);
checkPoolConnections(TEST_USER3, 2, 0);
{code}
this part need correction.
Here new ConnectionPoolId is created with TEST_USER2 but checkPoolConnections
is done using TEST_USER3.
In checkPoolConnections method
{code:java}
if (e.getKey().getUgi() == ugi)
{code}
then only it will validate numOfConns and numOfActiveConns. In this case for
TEST_USER3 ' *if* 'condition is returning *false* and if you pass any value
to the checkPoolConnections method, the test case will pass.
> TestConnectionManager#testCleanup() test case need correction
> -
>
> Key: HDFS-13289
> URL: https://issues.apache.org/jira/browse/HDFS-13289
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Dibyendu Karmakar
>Priority: Minor
>
> In TestConnectionManager#testCleanup()
>
> {code:java}
> // Make sure the number of connections doesn't go below minSize
> ConnectionPool pool3 = new ConnectionPool(
> conf, TEST_NN_ADDRESS, TEST_USER3, 2, 10);
> addConnectionsToPool(pool3, 10, 0);
> poolMap.put(new ConnectionPoolId(TEST_USER2, TEST_NN_ADDRESS), pool3);
> connManager.cleanup(pool3);
> checkPoolConnections(TEST_USER3, 2, 0);
> {code}
> this part need correction
> Here new ConnectionPoolId is created with TEST_USER2 but checkPoolConnections
> is done using TEST_USER3.
> In checkPoolConnections method
> {code:java}
> if (e.getKey().getUgi() == ugi)
> {code}
> then only it will validate numOfConns and numOfActiveConns. In this case for
> TEST_USER3 ' *if* 'condition is returning *false* and if you pass any value
> to the checkPoolConnections method, the test case will pass.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13289) TestConnectionManager#testCleanup() test case need correction
[
https://issues.apache.org/jira/browse/HDFS-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dibyendu Karmakar updated HDFS-13289:
-
Summary: TestConnectionManager#testCleanup() test case need correction
(was: TestConnectionManager test case need correction)
> TestConnectionManager#testCleanup() test case need correction
> -
>
> Key: HDFS-13289
> URL: https://issues.apache.org/jira/browse/HDFS-13289
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Dibyendu Karmakar
>Priority: Minor
>
> In TestConnectionManager#testCleanup()
>
> {code:java}
> // Make sure the number of connections doesn't go below minSize
> ConnectionPool pool3 = new ConnectionPool(
> conf, TEST_NN_ADDRESS, TEST_USER3, 2, 10);
> addConnectionsToPool(pool3, 10, 0);
> poolMap.put(new ConnectionPoolId(TEST_USER2, TEST_NN_ADDRESS), pool3);
> connManager.cleanup(pool3);
> checkPoolConnections(TEST_USER3, 2, 0);
> {code}
> this part need correction.
>
> Here new ConnectionPoolId is created with TEST_USER2 but checkPoolConnections
> is done using TEST_USER3.
> In checkPoolConnections method
> {code:java}
> if (e.getKey().getUgi() == ugi)
> {code}
> then only it will validate numOfConns and numOfActiveConns. In this case for
> TEST_USER3 ' *if* 'condition is returning *false* and if you pass any value
> to the checkPoolConnections method, the test case will pass.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13289) TestConnectionManager test case need correction
[
https://issues.apache.org/jira/browse/HDFS-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dibyendu Karmakar updated HDFS-13289:
-
Description:
In TestConnectionManager#testCleanup()
{code:java}
// Make sure the number of connections doesn't go below minSize
ConnectionPool pool3 = new ConnectionPool(
conf, TEST_NN_ADDRESS, TEST_USER3, 2, 10);
addConnectionsToPool(pool3, 10, 0);
poolMap.put(new ConnectionPoolId(TEST_USER2, TEST_NN_ADDRESS), pool3);
connManager.cleanup(pool3);
checkPoolConnections(TEST_USER3, 2, 0);
{code}
this part need correction.
Here new ConnectionPoolId is created with TEST_USER2 but checkPoolConnections
is done using TEST_USER3.
In checkPoolConnections method
{code:java}
if (e.getKey().getUgi() == ugi)
{code}
then only it will validate numOfConns and numOfActiveConns. In this case for
TEST_USER3 ' *if* 'condition is returning *false* and if you pass any value
to the checkPoolConnections method, the test case will pass.
was:
In TestConnectionManager.testCleanup()
{code:java}
// Make sure the number of connections doesn't go below minSize
ConnectionPool pool3 = new ConnectionPool(
conf, TEST_NN_ADDRESS, TEST_USER3, 2, 10);
addConnectionsToPool(pool3, 10, 0);
poolMap.put(new ConnectionPoolId(TEST_USER2, TEST_NN_ADDRESS), pool3);
connManager.cleanup(pool3);
checkPoolConnections(TEST_USER3, 2, 0);
{code}
this part need correction.
Here new ConnectionPoolId is created with TEST_USER2 but checkPoolConnections
is done using TEST_USER3.
In checkPoolConnections method
{code:java}
if (e.getKey().getUgi() == ugi)
{code}
then only it will validate numOfConns and numOfActiveConns. In this case for
TEST_USER3 ' *if* 'condition is returning *false* and if you pass any value
to the checkPoolConnections method, the test case will pass.
> TestConnectionManager test case need correction
> ---
>
> Key: HDFS-13289
> URL: https://issues.apache.org/jira/browse/HDFS-13289
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Dibyendu Karmakar
>Priority: Minor
>
> In TestConnectionManager#testCleanup()
>
> {code:java}
> // Make sure the number of connections doesn't go below minSize
> ConnectionPool pool3 = new ConnectionPool(
> conf, TEST_NN_ADDRESS, TEST_USER3, 2, 10);
> addConnectionsToPool(pool3, 10, 0);
> poolMap.put(new ConnectionPoolId(TEST_USER2, TEST_NN_ADDRESS), pool3);
> connManager.cleanup(pool3);
> checkPoolConnections(TEST_USER3, 2, 0);
> {code}
> this part need correction.
>
> Here new ConnectionPoolId is created with TEST_USER2 but checkPoolConnections
> is done using TEST_USER3.
> In checkPoolConnections method
> {code:java}
> if (e.getKey().getUgi() == ugi)
> {code}
> then only it will validate numOfConns and numOfActiveConns. In this case for
> TEST_USER3 ' *if* 'condition is returning *false* and if you pass any value
> to the checkPoolConnections method, the test case will pass.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13288) Why we don't add a harder lease expiration limit.
[
https://issues.apache.org/jira/browse/HDFS-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399841#comment-16399841
]
Vinayakumar B commented on HDFS-13288:
--
Namenode have a thread {{LeaseManager.Monitor}} checks for the leases which are
expired hard limit and triggers the lease recovery for those files.
If any exception in recovery of these lease, it (lease recovery from NN) should
repeat every one hour (hard limit).
Can you check in logs, whether this following log message available ?
{noformat}LOG.info("{} has expired hard limit", leaseToCheck); {noformat}
> Why we don't add a harder lease expiration limit.
> -
>
> Key: HDFS-13288
> URL: https://issues.apache.org/jira/browse/HDFS-13288
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.6.5
>Reporter: Igloo
>Priority: Minor
>
> Currently there exists a soft expire timeout(1 minutes by default) and hard
> expire timeout(60 minutes by default).
> On our production environment. Some client began writing a file long
> time(more than one year) ago, when writing finished and tried to close the
> output stream, the client failed closing it (for some IOException. etc. ).
> But the client process is a background service, it doesn't exit. So the lease
> doesn't released for more than one year.
> The problem is that, the lease for the file is occupied, we have to call
> recover lease on the file when doing demission or appending operation.
>
> So I am wondering why we don't add a more harder lease expire timeout, when a
> lease lasts too long (maybe one month), revoke it.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13288) Why we don't add a harder lease expiration limit.
[ https://issues.apache.org/jira/browse/HDFS-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igloo updated HDFS-13288: - Description: Currently there exists a soft expire timeout(1 minutes by default) and hard expire timeout(60 minutes by default). On our production environment. Some client began writing a file long time(more than one year) ago, when writing finished and tried to close the output stream, the client failed closing it (for some IOException. etc. ). But the client process is a background service, it doesn't exit. So the lease doesn't released for more than one year. The problem is that, the lease for the file is occupied, we have to call recover lease on the file when doing demission or appending operation. So I am wondering why we don't add a more harder lease expire timeout, when a lease lasts too long (maybe one month), revoke it. was: Currently there exists a soft expire timeout(1 minutes by default) and hard expire timeout(60 minutes by default). On our production environment. Some client began writing a file long time(more than one year) ago, when writing finished and tried to close the output stream, the client failed closing it (for some IOException. etc. ). But the client process is a background service, it doesn't exit. So the lease doesn't released for more than one year. The problem is the lease for the file is occupied, we have to call recover lease on the file. So I am wondering why we don't add a more harder lease expire timeout, when a lease lasts too long (maybe one month), revoke it. > Why we don't add a harder lease expiration limit. > - > > Key: HDFS-13288 > URL: https://issues.apache.org/jira/browse/HDFS-13288 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.6.5 >Reporter: Igloo >Priority: Minor > > Currently there exists a soft expire timeout(1 minutes by default) and hard > expire timeout(60 minutes by default). > On our production environment. Some client began writing a file long > time(more than one year) ago, when writing finished and tried to close the > output stream, the client failed closing it (for some IOException. etc. ). > But the client process is a background service, it doesn't exit. So the lease > doesn't released for more than one year. > The problem is that, the lease for the file is occupied, we have to call > recover lease on the file when doing demission or appending operation. > > So I am wondering why we don't add a more harder lease expire timeout, when a > lease lasts too long (maybe one month), revoke it. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13289) TestConnectionManager test case need correction
Dibyendu Karmakar created HDFS-13289:
Summary: TestConnectionManager test case need correction
Key: HDFS-13289
URL: https://issues.apache.org/jira/browse/HDFS-13289
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Dibyendu Karmakar
In TestConnectionManager.testCleanup()
{code:java}
// Make sure the number of connections doesn't go below minSize
ConnectionPool pool3 = new ConnectionPool(
conf, TEST_NN_ADDRESS, TEST_USER3, 2, 10);
addConnectionsToPool(pool3, 10, 0);
poolMap.put(new ConnectionPoolId(TEST_USER2, TEST_NN_ADDRESS), pool3);
connManager.cleanup(pool3);
checkPoolConnections(TEST_USER3, 2, 0);
{code}
this part need correction.
Here new ConnectionPoolId is created with TEST_USER2 but checkPoolConnections
is done using TEST_USER3.
In checkPoolConnections method
{code:java}
if (e.getKey().getUgi() == ugi)
{code}
then only it will validate numOfConns and numOfActiveConns. In this case for
TEST_USER3 ' *if* 'condition is returning *false* and if you pass any value
to the checkPoolConnections method, the test case will pass.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13261) Fix incorrect null value check
[
https://issues.apache.org/jira/browse/HDFS-13261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-13261:
-
Summary: Fix incorrect null value check (was: Incorrect null value check)
> Fix incorrect null value check
> --
>
> Key: HDFS-13261
> URL: https://issues.apache.org/jira/browse/HDFS-13261
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.3, 3.0.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Minor
> Attachments: HDFS-13261.001.patch, HDFS-13261.002-branch-2.patch,
> HDFS-13261.002.patch
>
>
>
> {code:java}
> public void setZoneName(final String name) {
> Preconditions.checkNotNull(name == null)---error parameter for
> checkNotNull
> zoneName = name;
> }
> {code}
> Like above, in 3.x, the checkNotNull function gets wrong parameter in two
> cases: \{{ZoneReencryptionStatus#setZoneName}} and
> \{{AdminHelper#HelpCommand#HelpCommand}}
> In 2.x, the error occurs once in \{{AdminHelper#HelpCommand#HelpCommand}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13288) Why we don't add a harder lease expiration limit.
[ https://issues.apache.org/jira/browse/HDFS-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igloo updated HDFS-13288: - Description: Currently there exists a soft expire timeout(1 minutes by default) and hard expire timeout(60 minutes by default). On our production environment. Some client began writing a file long time(more than one year) ago, when writing finished and tried to close the output stream, the client failed closing it (for some IOException. etc. ). But the client process is a background service, it doesn't exit. So the lease doesn't released for more than one year. The problem is the lease for the file is occupied, we have to call recover lease on the file. So I am wondering why we don't add a more harder lease expire timeout, when a lease lasts too long (maybe one month), revoke it. was: Currently there exists a soft expire timeout(1 minutes by default) and hard expire timeout(60 minutes by default). On our production environment. Some client began writing a file long time(more than one year) ago, when writing finished and tried to close the output stream, the client failed closing it (for some IOException. etc. ). But the client process is a background service, it doesn't exit. So the lease doesn't released for more than one year. The problem is the lease for the file is occupied, we have to call recover lease on the file. So I am wondering why we don't add a more harder lease expire timeout, when a lease lasts too long (maybe one month), revoke it. > Why we don't add a harder lease expiration limit. > - > > Key: HDFS-13288 > URL: https://issues.apache.org/jira/browse/HDFS-13288 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.6.5 >Reporter: Igloo >Priority: Minor > > Currently there exists a soft expire timeout(1 minutes by default) and hard > expire timeout(60 minutes by default). > On our production environment. Some client began writing a file long > time(more than one year) ago, when writing finished and tried to close the > output stream, the client failed closing it (for some IOException. etc. ). > But the client process is a background service, it doesn't exit. So the lease > doesn't released for more than one year. > The problem is the lease for the file is occupied, we have to call recover > lease on the file. > So I am wondering why we don't add a more harder lease expire timeout, when a > lease lasts too long (maybe one month), revoke it. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13261) Incorrect null value check
[
https://issues.apache.org/jira/browse/HDFS-13261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399834#comment-16399834
]
Yiqun Lin commented on HDFS-13261:
--
+1, committing...
> Incorrect null value check
> --
>
> Key: HDFS-13261
> URL: https://issues.apache.org/jira/browse/HDFS-13261
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.3, 3.0.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Minor
> Attachments: HDFS-13261.001.patch, HDFS-13261.002-branch-2.patch,
> HDFS-13261.002.patch
>
>
>
> {code:java}
> public void setZoneName(final String name) {
> Preconditions.checkNotNull(name == null)---error parameter for
> checkNotNull
> zoneName = name;
> }
> {code}
> Like above, in 3.x, the checkNotNull function gets wrong parameter in two
> cases: \{{ZoneReencryptionStatus#setZoneName}} and
> \{{AdminHelper#HelpCommand#HelpCommand}}
> In 2.x, the error occurs once in \{{AdminHelper#HelpCommand#HelpCommand}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13288) Why we don't add a harder lease expiration limit.
Igloo created HDFS-13288: Summary: Why we don't add a harder lease expiration limit. Key: HDFS-13288 URL: https://issues.apache.org/jira/browse/HDFS-13288 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.6.5 Reporter: Igloo Currently there exists a soft expire timeout(1 minutes by default) and hard expire timeout(60 minutes by default). On our production environment. Some client began writing a file long time(more than one year) ago, when writing finished and tried to close the output stream, the client failed closing it (for some IOException. etc. ). But the client process is a background service, it doesn't exit. So the lease doesn't released for more than one year. The problem is the lease for the file is occupied, we have to call recover lease on the file. So I am wondering why we don't add a more harder lease expire timeout, when a lease lasts too long (maybe one month), revoke it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12977) Add stateId to RPC headers.
[
https://issues.apache.org/jira/browse/HDFS-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399808#comment-16399808
]
Konstantin Shvachko edited comment on HDFS-12977 at 3/15/18 2:36 AM:
-
Great progress, Plamen. Few editorial comments
# {{AlignmentContext}} JavaDoc should say something like "Interface to align
the state between client and server via RPC communication".
#* Check for long lines
#* Needs annotations: {{@InterfaceAudience, @InterfaceStability}}
#* Abstract methods could use Javadoc too.
# {{ProtobufRpcEngine.Server}} Javadoc should say:
{code}
* @param alignmentContext provides server state info on client responses
{code}
#* The same in {{RpcEngine.getServer()}} and {{WritableRpcEngine.Server}}
# For {{RpcResponseHeaderProto}} we should probably just use {{stateId}} with
the comment that it is the last written global state id. In the response header
it is the last written GSI, but in request header it will become last seen id.
# {{GlobalStateIdContext}} needs Javadoc and annotations.
# {{TestClientTransactionId}} do we need to restart cluster for each test?
Would be better if start in {{@BeforeClass}} and shut down in {{@AfterClass}}
#* Very much like to see comments for test cases! It would be better to make
them Javadoc, and check for long lines.
# {{Client.setClientStateHandler()}} please update javadoc
#* {{DFSClient.lastSeenId}} -> {{lastSeenStateId}}
#* {{setClientStateHandler()}} should be updated to {{setAlignmentContext()}}
# How about {{ClientGCIContext}} instead of {{DfsClientAlignmentContext}}
#* needs Javadoc and annotations
# Also I recommend running checkstyle locally, I couldn't find artifacts on
Jenkins, but there may be diversions in the patch.
I think this completes the server part. We should think about setting GCI in
RequestHeader on the client side. In the next jira I assume.
was (Author: shv):
Great progress, Plamen. Few editorial comments
# {{AlignmentContext}} JavaDoc should say something like "Interface to align
the state between client and server via RPC communication".
#* Check for long lines
#* Needs annotations: {{@InterfaceAudience, @InterfaceStability}}
#* Abstract methods could use Javadoc too.
# {{ProtobufRpcEngine.Server}} Javadoc should say:
{code}
* @param alignmentContext provides server state info on client responses
{code}
#* The same in {{RpcEngine.getServer()}} and {{WritableRpcEngine.Server}}
# For {{RpcResponseHeaderProto}} we should probably just use {{stateId}} with
the comment that it is the last written global state id. In the response header
it is the last written GSI, but in request header it will become last seen id.
# {{GlobalStateIdContext}} needs Javadoc and annotations.
# {{TestClientTransactionId}} do we need to restart cluster for each test?
Would be better if start in {{@BeforeClass}} and shut down in {{@AfterClass}}
#* Very much like to see comments for test cases! It would be better to make
them Javadoc, and check for long lines.
# {{Client.setClientStateHandler()}} please update javadoc
#* {{DFSClient.lastSeenId}} -> {{lastSeenStateId}}
#* {{setClientStateHandler()}} should be updated to {{setAlignmentContext()}}
# How about {{ClientGCIContext}} instead of {{DfsClientAlignmentContext}}
#* needs Javadoc and annotations
I think this completes the server part. We should think about setting GCI in
RequestHeader on the client side. In the next jira I assume.
> Add stateId to RPC headers.
> ---
>
> Key: HDFS-12977
> URL: https://issues.apache.org/jira/browse/HDFS-12977
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ipc, namenode
>Reporter: Konstantin Shvachko
>Assignee: Plamen Jeliazkov
>Priority: Major
> Attachments: HDFS_12977.trunk.001.patch, HDFS_12977.trunk.002.patch,
> HDFS_12977.trunk.003.patch, HDFS_12977.trunk.004.patch,
> HDFS_12977.trunk.005.patch
>
>
> stateId is a new field in the RPC headers of NameNode proto calls.
> stateId is the journal transaction Id, which represents LastSeenId for the
> clients and LastWrittenId for NameNodes. See more in [reads from Standby
> design
> doc|https://issues.apache.org/jira/secure/attachment/12902925/ConsistentReadsFromStandbyNode.pdf].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12977) Add stateId to RPC headers.
[
https://issues.apache.org/jira/browse/HDFS-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399808#comment-16399808
]
Konstantin Shvachko commented on HDFS-12977:
Great progress, Plamen. Few editorial comments
# {{AlignmentContext}} JavaDoc should say something like "Interface to align
the state between client and server via RPC communication".
#* Check for long lines
#* Needs annotations: {{@InterfaceAudience, @InterfaceStability}}
#* Abstract methods could use Javadoc too.
# {{ProtobufRpcEngine.Server}} Javadoc should say:
{code}
* @param alignmentContext provides server state info on client responses
{code}
#* The same in {{RpcEngine.getServer()}} and {{WritableRpcEngine.Server}}
# For {{RpcResponseHeaderProto}} we should probably just use {{stateId}} with
the comment that it is the last written global state id. In the response header
it is the last written GSI, but in request header it will become last seen id.
# {{GlobalStateIdContext}} needs Javadoc and annotations.
# {{TestClientTransactionId}} do we need to restart cluster for each test?
Would be better if start in {{@BeforeClass}} and shut down in {{@AfterClass}}
#* Very much like to see comments for test cases! It would be better to make
them Javadoc, and check for long lines.
# {{Client.setClientStateHandler()}} please update javadoc
#* {{DFSClient.lastSeenId}} -> {{lastSeenStateId}}
#* {{setClientStateHandler()}} should be updated to {{setAlignmentContext()}}
# How about {{ClientGCIContext}} instead of {{DfsClientAlignmentContext}}
#* needs Javadoc and annotations
I think this completes the server part. We should think about setting GCI in
RequestHeader on the client side. In the next jira I assume.
> Add stateId to RPC headers.
> ---
>
> Key: HDFS-12977
> URL: https://issues.apache.org/jira/browse/HDFS-12977
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ipc, namenode
>Reporter: Konstantin Shvachko
>Assignee: Plamen Jeliazkov
>Priority: Major
> Attachments: HDFS_12977.trunk.001.patch, HDFS_12977.trunk.002.patch,
> HDFS_12977.trunk.003.patch, HDFS_12977.trunk.004.patch,
> HDFS_12977.trunk.005.patch
>
>
> stateId is a new field in the RPC headers of NameNode proto calls.
> stateId is the journal transaction Id, which represents LastSeenId for the
> clients and LastWrittenId for NameNodes. See more in [reads from Standby
> design
> doc|https://issues.apache.org/jira/secure/attachment/12902925/ConsistentReadsFromStandbyNode.pdf].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13275) Adding log for BlockPoolManager#refreshNamenodes failures
[ https://issues.apache.org/jira/browse/HDFS-13275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399031#comment-16399031 ] Ajay Kumar edited comment on HDFS-13275 at 3/15/18 2:34 AM: failed tests passed locally. [~xyao] thanks for review and commit. was (Author: ajayydv): failed tests passed locally. > Adding log for BlockPoolManager#refreshNamenodes failures > - > > Key: HDFS-13275 > URL: https://issues.apache.org/jira/browse/HDFS-13275 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Minor > Labels: logs > Fix For: 3.1.0, 3.2.0 > > Attachments: HDFS-13275.000.patch > > > This helps detect and troubleshooting misconfigured NN service RPC or NN > lifeline RPC address. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13287) TestINodeFile#testGetBlockType results in NPE when run alone
[
https://issues.apache.org/jira/browse/HDFS-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399807#comment-16399807
]
genericqa commented on HDFS-13287:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 15s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-13287 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12914605/HDFS-13287.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 299298489e7e 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e840b4a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23495/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23495/testReport/ |
| Max. process+thread count | 3405 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399799#comment-16399799 ] Íñigo Goiri commented on HDFS-13215: At this point, I think the only patches close to completion are HDFS-13204 and HDFS-13224. HDFS-13248 is the other one with a patch available but there is no rush for this one as it will take me a few days to code. The rest are still in discussion. Anyway, we can do a freeze whenever we need. > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13224) RBF: Resolvers to support mount points across multiple subclusters
[ https://issues.apache.org/jira/browse/HDFS-13224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399800#comment-16399800 ] Yiqun Lin commented on HDFS-13224: -- LGTM, +1. Thanks [~elgoiri]. You may attach a clean patch in HDFS-13250 after this commit and I will take a review today, :). > RBF: Resolvers to support mount points across multiple subclusters > -- > > Key: HDFS-13224 > URL: https://issues.apache.org/jira/browse/HDFS-13224 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13224.000.patch, HDFS-13224.001.patch, > HDFS-13224.002.patch, HDFS-13224.003.patch, HDFS-13224.004.patch, > HDFS-13224.005.patch, HDFS-13224.006.patch, HDFS-13224.007.patch, > HDFS-13224.008.patch, HDFS-13224.009.patch, HDFS-13224.010.patch > > > Currently, a mount point points to a single subcluster. We should be able to > spread files in a mount point across subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13055) Aggregate usage statistics from datanodes
[
https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399794#comment-16399794
]
Ajay Kumar commented on HDFS-13055:
---
[~arpitagarwal] thanks, patch v7 contains first part (i.e Generate usage report
on DN.). Will create child jiras and upload patch for remaining 2 steps.
> Aggregate usage statistics from datanodes
> -
>
> Key: HDFS-13055
> URL: https://issues.apache.org/jira/browse/HDFS-13055
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-13055.001.patch, HDFS-13055.002.patch,
> HDFS-13055.003.patch, HDFS-13055.004.patch, HDFS-13055.005.patch,
> HDFS-13055.006.patch, HDFS-13055.007.patch
>
>
> We collect variety of statistics in DataNodes and expose them via JMX.
> Aggregating some of the high level statistics which we are already collecting
> in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable
> time window will create a central repository accessible via JMX and UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[
https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399793#comment-16399793
]
Íñigo Goiri commented on HDFS-13215:
{quote}
Given that this Jira will new a module and make a big refactor, i suggest we
should finish this Jira as soon as possible, so that the new Jira ticket will
base on the new layout.
BTW, Not sure if we should suspend a few day of commit for RBF, if not Wei Yan
will have to update for getting the newer commit
{quote}
I agree.
In addition, doing this for all the branches is even more convoluted.
At this point, I think [~ywskycn] is still trying to figure out what's the best
split.
Once we have a clear approach, we should freeze RBF commits until done.
[~ywskycn], how far you think you are from having a full approach?
> RBF: Move Router to its own module
> --
>
> Key: HDFS-13215
> URL: https://issues.apache.org/jira/browse/HDFS-13215
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Wei Yan
>Priority: Major
> Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch,
> HDFS-13215.002.patch, HDFS-13215.003.patch
>
>
> We are splitting the HDFS client code base and potentially Router-based
> Federation is also independent enough to be in its own package.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[
https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HDFS-13055:
--
Attachment: HDFS-13055.007.patch
> Aggregate usage statistics from datanodes
> -
>
> Key: HDFS-13055
> URL: https://issues.apache.org/jira/browse/HDFS-13055
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-13055.001.patch, HDFS-13055.002.patch,
> HDFS-13055.003.patch, HDFS-13055.004.patch, HDFS-13055.005.patch,
> HDFS-13055.006.patch, HDFS-13055.007.patch
>
>
> We collect variety of statistics in DataNodes and expose them via JMX.
> Aggregating some of the high level statistics which we are already collecting
> in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable
> time window will create a central repository accessible via JMX and UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13088) Allow HDFS files/blocks to be over-replicated.
[ https://issues.apache.org/jira/browse/HDFS-13088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399788#comment-16399788 ] Íñigo Goiri commented on HDFS-13088: I guess there is no way to add new fields to the header. I think keeping backwards compatibility is pretty important. In addition, we should support over replications higher than 8. > Allow HDFS files/blocks to be over-replicated. > -- > > Key: HDFS-13088 > URL: https://issues.apache.org/jira/browse/HDFS-13088 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti >Priority: Major > Attachments: HDFS-13088.001.patch > > > This JIRA is to add a per-file "over-replication" factor to HDFS. As > mentioned in HDFS-13069, the over-replication factor will be the excess > replicas that will be allowed to exist for a file or block. This is > beneficial if the application deems additional replicas for a file are > needed. In the case of HDFS-13069, it would allow copies of data in PROVIDED > storage to be cached locally in HDFS in a read-through manner. > The Namenode will not proactively meet the over-replication i.e., it does not > schedule replications if the number of replicas for a block is less than > (replication factor + over-replication factor) as long as they are more than > the replication factor of the file. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[
https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399780#comment-16399780
]
genericqa commented on HDFS-13215:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 71 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdfs-project . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
32s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
49s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
26s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
40s{color} | {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 5m
9s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 2m
22s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 2m 22s{color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 22s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 12s{color} | {color:orange} root: The patch generated 3 new + 432 unchanged
- 3 fixed = 435 total (was 435) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 6m
25s{color} | {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
10s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
57s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdfs-project . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
33s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc
[jira] [Comment Edited] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399764#comment-16399764 ] maobaolong edited comment on HDFS-13215 at 3/15/18 1:51 AM: Given that this Jira will new a module and make a big refactor, i suggest we should finish this Jira as soon as possible, so that the new Jira ticket will base on the new layout. BTW, Not sure if we should suspend a few day of commit for RBF, if not [~ywskycn] will have to update for getting the newer commit was (Author: maobaolong): Given that this Jira will new a module and make a big refactor, i suggest we should finish this Jira as soon as possible, so that the new Jira ticket will base on the new layout. > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13246) FileInputStream redundant closes in readReplicasFromCache
[
https://issues.apache.org/jira/browse/HDFS-13246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399762#comment-16399762
]
liaoyuxiangqin commented on HDFS-13246:
---
[~xiaochen] Can you help me review this patch, thanks.
> FileInputStream redundant closes in readReplicasFromCache
> --
>
> Key: HDFS-13246
> URL: https://issues.apache.org/jira/browse/HDFS-13246
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: liaoyuxiangqin
>Priority: Major
> Attachments: HDFS-13246.001.patch
>
>
> When i read the readReplicasFromCache() of BlockPoolSlice class in datanode,
> I found the following code closes fileinputstream redundant, I think
> IOUtils.closeStream(inputStream) in finally code block could guarantee close
> the inputStream correctly, So the
> inputStream.close() can remove. Thanks.
>
> {code:java|title=BlockPoolSlice.java|borderStyle=solid}
> FileInputStream inputStream = null;
> try {
> inputStream = fileIoProvider.getFileInputStream(volume, replicaFile);
> BlockListAsLongs blocksList =
> BlockListAsLongs.readFrom(inputStream, maxDataLength);
> if (blocksList == null) {
> return false;
> }
> for (BlockReportReplica replica : blocksList) {
> switch (replica.getState()) {
> case FINALIZED:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, true);
> break;
> case RUR:
> case RBW:
> case RWR:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, false);
> break;
> default:
> break;
> }
> }
> inputStream.close();
> // Now it is safe to add the replica into volumeMap
> // In case of any exception during parsing this cache file, fall back
> // to scan all the files on disk.
> for (Iterator iter =
> tmpReplicaMap.replicas(bpid).iterator(); iter.hasNext(); ) {
> ReplicaInfo info = iter.next();
> // We use a lightweight GSet to store replicaInfo, we need to remove
> // it from one GSet before adding to another.
> iter.remove();
> volumeMap.add(bpid, info);
> }
> LOG.info("Successfully read replica from cache file : "
> + replicaFile.getPath());
> return true;
> } catch (Exception e) {
> // Any exception we need to revert back to read from disk
> // Log the error and return false
> LOG.info("Exception occurred while reading the replicas cache file: "
> + replicaFile.getPath(), e );
> return false;
> }
> finally {
> if (!fileIoProvider.delete(volume, replicaFile)) {
> LOG.info("Failed to delete replica cache file: " +
> replicaFile.getPath());
> }
> // close the inputStream
> IOUtils.closeStream(inputStream);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399764#comment-16399764 ] maobaolong commented on HDFS-13215: --- Given that this Jira will new a module and make a big refactor, i suggest we should finish this Jira as soon as possible, so that the new Jira ticket will base on the new layout. > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[
https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399761#comment-16399761
]
genericqa commented on HDFS-13215:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 98 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 13m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdfs-project . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
0s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
34s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
52s{color} | {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
37s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 5m
47s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 2m
22s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 2m 22s{color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 22s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 41s{color} | {color:orange} root: The patch generated 17 new + 647 unchanged
- 14 fixed = 664 total (was 661) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 7m
27s{color} | {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
1s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
9s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 7 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
7s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
55s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdfs-project . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
38s{color} |
[jira] [Commented] (HDFS-13246) FileInputStream redundant closes in readReplicasFromCache
[
https://issues.apache.org/jira/browse/HDFS-13246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399756#comment-16399756
]
liaoyuxiangqin commented on HDFS-13246:
---
[~shahrs87] Thanks for your review on this.
> FileInputStream redundant closes in readReplicasFromCache
> --
>
> Key: HDFS-13246
> URL: https://issues.apache.org/jira/browse/HDFS-13246
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: liaoyuxiangqin
>Priority: Major
> Attachments: HDFS-13246.001.patch
>
>
> When i read the readReplicasFromCache() of BlockPoolSlice class in datanode,
> I found the following code closes fileinputstream redundant, I think
> IOUtils.closeStream(inputStream) in finally code block could guarantee close
> the inputStream correctly, So the
> inputStream.close() can remove. Thanks.
>
> {code:java|title=BlockPoolSlice.java|borderStyle=solid}
> FileInputStream inputStream = null;
> try {
> inputStream = fileIoProvider.getFileInputStream(volume, replicaFile);
> BlockListAsLongs blocksList =
> BlockListAsLongs.readFrom(inputStream, maxDataLength);
> if (blocksList == null) {
> return false;
> }
> for (BlockReportReplica replica : blocksList) {
> switch (replica.getState()) {
> case FINALIZED:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, true);
> break;
> case RUR:
> case RBW:
> case RWR:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, false);
> break;
> default:
> break;
> }
> }
> inputStream.close();
> // Now it is safe to add the replica into volumeMap
> // In case of any exception during parsing this cache file, fall back
> // to scan all the files on disk.
> for (Iterator iter =
> tmpReplicaMap.replicas(bpid).iterator(); iter.hasNext(); ) {
> ReplicaInfo info = iter.next();
> // We use a lightweight GSet to store replicaInfo, we need to remove
> // it from one GSet before adding to another.
> iter.remove();
> volumeMap.add(bpid, info);
> }
> LOG.info("Successfully read replica from cache file : "
> + replicaFile.getPath());
> return true;
> } catch (Exception e) {
> // Any exception we need to revert back to read from disk
> // Log the error and return false
> LOG.info("Exception occurred while reading the replicas cache file: "
> + replicaFile.getPath(), e );
> return false;
> }
> finally {
> if (!fileIoProvider.delete(volume, replicaFile)) {
> LOG.info("Failed to delete replica cache file: " +
> replicaFile.getPath());
> }
> // close the inputStream
> IOUtils.closeStream(inputStream);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13219) RBF: Cluster information on Router is not correct when the Federation shares datanodes
[
https://issues.apache.org/jira/browse/HDFS-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399755#comment-16399755
]
Íñigo Goiri commented on HDFS-13219:
Thanks [~maobaolong] for putting examples.
In my comment I already agreed that this was an issue and proposed a solution.
For solving this correctly, we need to use {{getDatanodeReport()}}; this will
give us all the data and then we can aggregate.
However, this is pretty expensive so we should:
* Cache this information
* Have an option to use the old and the new approach
> RBF: Cluster information on Router is not correct when the Federation shares
> datanodes
> --
>
> Key: HDFS-13219
> URL: https://issues.apache.org/jira/browse/HDFS-13219
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.9.0
>Reporter: Tao Jie
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png,
> screenshot-4.png, screenshot-5.png
>
>
> Now summary information on Router website aggregates summary of each
> nameservice. However in a typical federation cluster deployment, datanodes
> are shared among nameservices. Consider we have 2 namespaces and 100
> datanodes in one cluster. 100 datanodes are available for each namespace, but
> we see 200 datanodes on the router website. So does other information such as
> {{Total capacity}}, {{Remaining capacity}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13217) Log audit event only used last EC policy name when add multiple policies from file
[
https://issues.apache.org/jira/browse/HDFS-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384486#comment-16384486
]
liaoyuxiangqin edited comment on HDFS-13217 at 3/15/18 1:40 AM:
[~eddyxu] Thank you for your attention on this, i have submit a original patch,
Thanks!
was (Author: liaoyuxiangqin):
[~eddyxu] Thank you for your attention on this, i have submit a patch, Thanks!
> Log audit event only used last EC policy name when add multiple policies from
> file
> ---
>
> Key: HDFS-13217
> URL: https://issues.apache.org/jira/browse/HDFS-13217
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: liaoyuxiangqin
>Assignee: liaoyuxiangqin
>Priority: Major
> Attachments: HDFS-13217.001.patch, HDFS-13217.002.patch
>
>
> When i read the addErasureCodingPolicies() of FSNamesystem class in namenode,
> i found the following code only used last ec policy name for logAuditEvent,
> i think this audit log can't track whole policies for the add multiple
> erasure coding policies to the ErasureCodingPolicyManager. Thanks.
> {code:java|title=FSNamesystem.java|borderStyle=solid}
> try {
> checkOperation(OperationCategory.WRITE);
> checkNameNodeSafeMode("Cannot add erasure coding policy");
> for (ErasureCodingPolicy policy : policies) {
> try {
> ErasureCodingPolicy newPolicy =
> FSDirErasureCodingOp.addErasureCodingPolicy(this, policy,
> logRetryCache);
> addECPolicyName = newPolicy.getName();
> responses.add(new AddErasureCodingPolicyResponse(newPolicy));
> } catch (HadoopIllegalArgumentException e) {
> responses.add(new AddErasureCodingPolicyResponse(policy, e));
> }
> }
> success = true;
> return responses.toArray(new AddErasureCodingPolicyResponse[0]);
> } finally {
> writeUnlock(operationName);
> if (success) {
> getEditLog().logSync();
> }
> logAuditEvent(success, operationName,addECPolicyName, null, null);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13217) Log audit event only used last EC policy name when add multiple policies from file
[
https://issues.apache.org/jira/browse/HDFS-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384486#comment-16384486
]
liaoyuxiangqin edited comment on HDFS-13217 at 3/15/18 1:39 AM:
[~eddyxu] Thank you for your attention on this, i have submit a patch, Thanks!
was (Author: liaoyuxiangqin):
[~eddyxu] Thank you for your attention on this, i have submit a patch. Thanks!
> Log audit event only used last EC policy name when add multiple policies from
> file
> ---
>
> Key: HDFS-13217
> URL: https://issues.apache.org/jira/browse/HDFS-13217
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: liaoyuxiangqin
>Assignee: liaoyuxiangqin
>Priority: Major
> Attachments: HDFS-13217.001.patch, HDFS-13217.002.patch
>
>
> When i read the addErasureCodingPolicies() of FSNamesystem class in namenode,
> i found the following code only used last ec policy name for logAuditEvent,
> i think this audit log can't track whole policies for the add multiple
> erasure coding policies to the ErasureCodingPolicyManager. Thanks.
> {code:java|title=FSNamesystem.java|borderStyle=solid}
> try {
> checkOperation(OperationCategory.WRITE);
> checkNameNodeSafeMode("Cannot add erasure coding policy");
> for (ErasureCodingPolicy policy : policies) {
> try {
> ErasureCodingPolicy newPolicy =
> FSDirErasureCodingOp.addErasureCodingPolicy(this, policy,
> logRetryCache);
> addECPolicyName = newPolicy.getName();
> responses.add(new AddErasureCodingPolicyResponse(newPolicy));
> } catch (HadoopIllegalArgumentException e) {
> responses.add(new AddErasureCodingPolicyResponse(policy, e));
> }
> }
> success = true;
> return responses.toArray(new AddErasureCodingPolicyResponse[0]);
> } finally {
> writeUnlock(operationName);
> if (success) {
> getEditLog().logSync();
> }
> logAuditEvent(success, operationName,addECPolicyName, null, null);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13219) RBF: Cluster information on Router is not correct when the Federation shares datanodes
[
https://issues.apache.org/jira/browse/HDFS-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399738#comment-16399738
]
maobaolong commented on HDFS-13219:
---
Maybe we can see something wrong from the following wrong:
- ns1-nn1-datanode
!screenshot-4.png!
ns10-nn20-datanode
!screenshot-5.png!
- ns-fed-r1-overview
!screenshot-3.png!
- ns-fed-r1-subcluster
!screenshot-2.png!
- ns-fed-r1-datanode
!screenshot-1.png!
We can see something wrong:
* Router overview page should not display the live datanode to 8, because we
have 4 datanode in all.
* Router datanode page should not display the ns1's datanode information,
because we have 2 ns (ns1 and ns10).
[~linyiqun] [~elgoiri], Please take a look at this issue, maybe we should
discuss for it.
> RBF: Cluster information on Router is not correct when the Federation shares
> datanodes
> --
>
> Key: HDFS-13219
> URL: https://issues.apache.org/jira/browse/HDFS-13219
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.9.0
>Reporter: Tao Jie
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png,
> screenshot-4.png, screenshot-5.png
>
>
> Now summary information on Router website aggregates summary of each
> nameservice. However in a typical federation cluster deployment, datanodes
> are shared among nameservices. Consider we have 2 namespaces and 100
> datanodes in one cluster. 100 datanodes are available for each namespace, but
> we see 200 datanodes on the router website. So does other information such as
> {{Total capacity}}, {{Remaining capacity}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13088) Allow HDFS files/blocks to be over-replicated.
[
https://issues.apache.org/jira/browse/HDFS-13088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399734#comment-16399734
]
Virajith Jalaparti commented on HDFS-13088:
---
[~elgoiri] - An alternative would be to create a new field in {{InodeFile}} for
the over-replication factor instead of changing the {{HeaderFormat}}.
> Allow HDFS files/blocks to be over-replicated.
> --
>
> Key: HDFS-13088
> URL: https://issues.apache.org/jira/browse/HDFS-13088
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HDFS-13088.001.patch
>
>
> This JIRA is to add a per-file "over-replication" factor to HDFS. As
> mentioned in HDFS-13069, the over-replication factor will be the excess
> replicas that will be allowed to exist for a file or block. This is
> beneficial if the application deems additional replicas for a file are
> needed. In the case of HDFS-13069, it would allow copies of data in PROVIDED
> storage to be cached locally in HDFS in a read-through manner.
> The Namenode will not proactively meet the over-replication i.e., it does not
> schedule replications if the number of replicas for a block is less than
> (replication factor + over-replication factor) as long as they are more than
> the replication factor of the file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12723) TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing consistently.
[
https://issues.apache.org/jira/browse/HDFS-12723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399731#comment-16399731
]
Íñigo Goiri commented on HDFS-12723:
The Yetus run for [^HDFS-12723.002.patch] was clean and after testing locally
the unit test passes consistently.
Even though this test is pretty heavy and slow, this will be an improvement and
will remove one spurious failed unit test.
+1
If nobody is against, I'll commit this tomorrow.
Meanwhile, we can work on making it leaner in HDFS-13285.
> TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing
> consistently.
>
>
> Key: HDFS-12723
> URL: https://issues.apache.org/jira/browse/HDFS-12723
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Rushabh S Shah
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-12723.000.patch, HDFS-12723.001.patch,
> HDFS-12723.002.patch
>
>
> TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks is timing
> out consistently on my local machine.
> {noformat}
> Running org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 132.405 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
> testReadFileWithMissingBlocks(org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks)
> Time elapsed: 132.171 sec <<< ERROR!
> java.util.concurrent.TimeoutException: Timed out waiting for /foo to have all
> the internalBlocks
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:295)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:256)
> at
> org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.readFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:98)
> at
> org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:82)
> Results :
> Tests in error:
>
> TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks:82->readFileWithMissingBlocks:98
> » Timeout
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13088) Allow HDFS files/blocks to be over-replicated.
[
https://issues.apache.org/jira/browse/HDFS-13088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399725#comment-16399725
]
Íñigo Goiri commented on HDFS-13088:
Thanks [~virajith] for working on this.
We definitely see value for this feature once it's combined with tiered storage
and caching.
The most risky change here is the {{INodeFile}} change.
If there are files with replication >=256, the NameNode will fail to start.
This was also tweaked in HDFS-7866 for EC.
Is there a way we can add the over replication without restricting the number
of replicas?
Note that we would only allow 8 replicas for over replication.
> Allow HDFS files/blocks to be over-replicated.
> --
>
> Key: HDFS-13088
> URL: https://issues.apache.org/jira/browse/HDFS-13088
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HDFS-13088.001.patch
>
>
> This JIRA is to add a per-file "over-replication" factor to HDFS. As
> mentioned in HDFS-13069, the over-replication factor will be the excess
> replicas that will be allowed to exist for a file or block. This is
> beneficial if the application deems additional replicas for a file are
> needed. In the case of HDFS-13069, it would allow copies of data in PROVIDED
> storage to be cached locally in HDFS in a read-through manner.
> The Namenode will not proactively meet the over-replication i.e., it does not
> schedule replications if the number of replicas for a block is less than
> (replication factor + over-replication factor) as long as they are more than
> the replication factor of the file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13219) RBF: Cluster information on Router is not correct when the Federation shares datanodes
[
https://issues.apache.org/jira/browse/HDFS-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13219:
--
Attachment: screenshot-5.png
> RBF: Cluster information on Router is not correct when the Federation shares
> datanodes
> --
>
> Key: HDFS-13219
> URL: https://issues.apache.org/jira/browse/HDFS-13219
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.9.0
>Reporter: Tao Jie
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png,
> screenshot-4.png, screenshot-5.png
>
>
> Now summary information on Router website aggregates summary of each
> nameservice. However in a typical federation cluster deployment, datanodes
> are shared among nameservices. Consider we have 2 namespaces and 100
> datanodes in one cluster. 100 datanodes are available for each namespace, but
> we see 200 datanodes on the router website. So does other information such as
> {{Total capacity}}, {{Remaining capacity}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12723) TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing consistently.
[
https://issues.apache.org/jira/browse/HDFS-12723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399716#comment-16399716
]
genericqa commented on HDFS-12723:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 204 unchanged - 1 fixed = 204 total (was 205) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestReconstructStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-12723 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12914594/HDFS-12723.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux c1b8e9fc2d86 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e840b4a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23492/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23492/testReport/ |
| Max. process+thread count | 3098 (vs. ulimit of 1) |
| modules | C:
[jira] [Commented] (HDFS-13258) Ozone: restructure Hdsl/Ozone code to separated maven subprojects
[
https://issues.apache.org/jira/browse/HDFS-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399710#comment-16399710
]
genericqa commented on HDFS-13258:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 266 new or modified
test files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 9m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
46s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
24s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
25s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-dist . hadoop-tools/hadoop-tools-dist {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
11s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
HDFS-7240 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
45s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 32m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 13m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 13m 22s{color}
| {color:red} root generated 4 new + 1284 unchanged - 17 fixed = 1288 total
(was 1301) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 25s{color} | {color:orange} root: The patch generated 76 new + 1403
unchanged - 37 fixed = 1479 total (was 1440) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 16m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
38s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:orange}-0{color} | {color:orange} shelldocs {color} | {color:orange}
0m 9s{color} | {color:orange} The patch generated 4 new + 100 unchanged - 0
fixed = 104 total (was 100) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
2s{color} | {color:red} The patch has 13 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
2s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
40s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 35s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-hdsl hadoop-ozone hadoop-cblock . hadoop-dist
hadoop-ozone/integration-test hadoop-tools/hadoop-tools-dist {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 21m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 7m
[jira] [Comment Edited] (HDFS-13088) Allow HDFS files/blocks to be over-replicated.
[
https://issues.apache.org/jira/browse/HDFS-13088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399706#comment-16399706
]
Virajith Jalaparti edited comment on HDFS-13088 at 3/15/18 1:11 AM:
Posting an initial patch to get feedback on the approach. The key changes are:
1) Add a new {{setReplication(string, short, short)}} and
{{getOverReplication}} calls to {{FileSystem}}.
2) Change the {{InodeFile#HeaderFormat}} so that of the 11bits that were
reserved for the replication factor, 3 bits are used for over-replication. This
implies that the maximum allowed replication will become 2^8 -1 instead of 2^11
-1.
3) Change the {{setrep}} command (and ClientNamenodeProtocol) to allow setting
the over-replication factor on a file.
The idea behind the changes was that over-replication is a "new kind" of
replication factor, and thus, I modified existing ways to set the replication
factor on a file to include over-replication
was (Author: virajith):
Posting an initial patch to get feedback on the approach. The key changes are:
1) Add a new {{setReplication(string, short, short)}} and
{{getOverReplication}} calls to {{FileSystem}}.
2) Change the {{InodeFile#HeaderFormat}} so that of the 11bits that were
reserved for the replication factor, 3 bits are used for over-replication
3) Change the {{setrep}} command (and ClientNamenodeProtocol) to allow setting
the over-replication factor on a file.
The idea behind the changes was that over-replication is a "new kind" of
replication factor, and thus, I modified existing ways to set the replication
factor on a file to include over-replication
> Allow HDFS files/blocks to be over-replicated.
> --
>
> Key: HDFS-13088
> URL: https://issues.apache.org/jira/browse/HDFS-13088
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HDFS-13088.001.patch
>
>
> This JIRA is to add a per-file "over-replication" factor to HDFS. As
> mentioned in HDFS-13069, the over-replication factor will be the excess
> replicas that will be allowed to exist for a file or block. This is
> beneficial if the application deems additional replicas for a file are
> needed. In the case of HDFS-13069, it would allow copies of data in PROVIDED
> storage to be cached locally in HDFS in a read-through manner.
> The Namenode will not proactively meet the over-replication i.e., it does not
> schedule replications if the number of replicas for a block is less than
> (replication factor + over-replication factor) as long as they are more than
> the replication factor of the file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13088) Allow HDFS files/blocks to be over-replicated.
[
https://issues.apache.org/jira/browse/HDFS-13088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399706#comment-16399706
]
Virajith Jalaparti commented on HDFS-13088:
---
Posting an initial patch to get feedback on the approach. The key changes are:
1) Add a new {{setReplication(string, short, short)}} and
{{getOverReplication}} calls to {{FileSystem}}.
2) Change the {{InodeFile#HeaderFormat}} so that of the 11bits that were
reserved for the replication factor, 3 bits are used for over-replication
3) Change the {{setrep}} command (and ClientNamenodeProtocol) to allow setting
the over-replication factor on a file.
The idea behind the changes was that over-replication is a "new kind" of
replication factor, and thus, I modified existing ways to set the replication
factor on a file to include over-replication
> Allow HDFS files/blocks to be over-replicated.
> --
>
> Key: HDFS-13088
> URL: https://issues.apache.org/jira/browse/HDFS-13088
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HDFS-13088.001.patch
>
>
> This JIRA is to add a per-file "over-replication" factor to HDFS. As
> mentioned in HDFS-13069, the over-replication factor will be the excess
> replicas that will be allowed to exist for a file or block. This is
> beneficial if the application deems additional replicas for a file are
> needed. In the case of HDFS-13069, it would allow copies of data in PROVIDED
> storage to be cached locally in HDFS in a read-through manner.
> The Namenode will not proactively meet the over-replication i.e., it does not
> schedule replications if the number of replicas for a block is less than
> (replication factor + over-replication factor) as long as they are more than
> the replication factor of the file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13219) RBF: Cluster information on Router is not correct when the Federation shares datanodes
[
https://issues.apache.org/jira/browse/HDFS-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13219:
--
Attachment: screenshot-4.png
> RBF: Cluster information on Router is not correct when the Federation shares
> datanodes
> --
>
> Key: HDFS-13219
> URL: https://issues.apache.org/jira/browse/HDFS-13219
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.9.0
>Reporter: Tao Jie
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png,
> screenshot-4.png
>
>
> Now summary information on Router website aggregates summary of each
> nameservice. However in a typical federation cluster deployment, datanodes
> are shared among nameservices. Consider we have 2 namespaces and 100
> datanodes in one cluster. 100 datanodes are available for each namespace, but
> we see 200 datanodes on the router website. So does other information such as
> {{Total capacity}}, {{Remaining capacity}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13088) Allow HDFS files/blocks to be over-replicated.
[ https://issues.apache.org/jira/browse/HDFS-13088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti reassigned HDFS-13088: - Assignee: Virajith Jalaparti > Allow HDFS files/blocks to be over-replicated. > -- > > Key: HDFS-13088 > URL: https://issues.apache.org/jira/browse/HDFS-13088 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti >Priority: Major > Attachments: HDFS-13088.001.patch > > > This JIRA is to add a per-file "over-replication" factor to HDFS. As > mentioned in HDFS-13069, the over-replication factor will be the excess > replicas that will be allowed to exist for a file or block. This is > beneficial if the application deems additional replicas for a file are > needed. In the case of HDFS-13069, it would allow copies of data in PROVIDED > storage to be cached locally in HDFS in a read-through manner. > The Namenode will not proactively meet the over-replication i.e., it does not > schedule replications if the number of replicas for a block is less than > (replication factor + over-replication factor) as long as they are more than > the replication factor of the file. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13284) Adjust criteria for LowRedundancyBlocks.QUEUE_VERY_LOW_REDUNDANCY
[ https://issues.apache.org/jira/browse/HDFS-13284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399574#comment-16399574 ] Íñigo Goiri edited comment on HDFS-13284 at 3/15/18 1:07 AM: - [~lukmajercak], could you give a high level description? This code is pretty old. [~ste...@apache.org], could you take a look? was (Author: elgoiri): [~lukmajercak], could you give a high levle description? This code is pretty old. [~ste...@apache.org], could you take a look? > Adjust criteria for LowRedundancyBlocks.QUEUE_VERY_LOW_REDUNDANCY > - > > Key: HDFS-13284 > URL: https://issues.apache.org/jira/browse/HDFS-13284 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, namenode >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HDFS-13284.000.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12422) Replace DataNode in Pipeline when waiting for Last Packet fails
[ https://issues.apache.org/jira/browse/HDFS-12422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399703#comment-16399703 ] Íñigo Goiri commented on HDFS-12422: Another piece of code that comes from a long time ago. [~chris.douglas], do you know anybody fit for reviewing this? We had this issue a while back from customers not closing their streams. > Replace DataNode in Pipeline when waiting for Last Packet fails > --- > > Key: HDFS-12422 > URL: https://issues.apache.org/jira/browse/HDFS-12422 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs, hdfs-client >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Labels: hdfs > Attachments: HDFS-12422.001.patch, HDFS-12422.002.patch > > > # Create a file with replicationFactor = 4, minReplicas = 2 > # Fail waiting for the last packet, followed by 2 exceptions when recovering > the leftover pipeline > # The leftover pipeline will only have one DN and NN will never close such > block, resulting in failure to write > The block will stay there forever, unable to be replicated, ultimately going > missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13219) RBF: Cluster information on Router is not correct when the Federation shares datanodes
[
https://issues.apache.org/jira/browse/HDFS-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13219:
--
Attachment: screenshot-3.png
> RBF: Cluster information on Router is not correct when the Federation shares
> datanodes
> --
>
> Key: HDFS-13219
> URL: https://issues.apache.org/jira/browse/HDFS-13219
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.9.0
>Reporter: Tao Jie
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> Now summary information on Router website aggregates summary of each
> nameservice. However in a typical federation cluster deployment, datanodes
> are shared among nameservices. Consider we have 2 namespaces and 100
> datanodes in one cluster. 100 datanodes are available for each namespace, but
> we see 200 datanodes on the router website. So does other information such as
> {{Total capacity}}, {{Remaining capacity}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13088) Allow HDFS files/blocks to be over-replicated.
[ https://issues.apache.org/jira/browse/HDFS-13088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-13088: -- Attachment: HDFS-13088.001.patch > Allow HDFS files/blocks to be over-replicated. > -- > > Key: HDFS-13088 > URL: https://issues.apache.org/jira/browse/HDFS-13088 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Priority: Major > Attachments: HDFS-13088.001.patch > > > This JIRA is to add a per-file "over-replication" factor to HDFS. As > mentioned in HDFS-13069, the over-replication factor will be the excess > replicas that will be allowed to exist for a file or block. This is > beneficial if the application deems additional replicas for a file are > needed. In the case of HDFS-13069, it would allow copies of data in PROVIDED > storage to be cached locally in HDFS in a read-through manner. > The Namenode will not proactively meet the over-replication i.e., it does not > schedule replications if the number of replicas for a block is less than > (replication factor + over-replication factor) as long as they are more than > the replication factor of the file. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13224) RBF: Resolvers to support mount points across multiple subclusters
[ https://issues.apache.org/jira/browse/HDFS-13224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399699#comment-16399699 ] Íñigo Goiri commented on HDFS-13224: The unit tests now are fine. [~ywskycn], [~linyiqun], do you mind reviewing this? > RBF: Resolvers to support mount points across multiple subclusters > -- > > Key: HDFS-13224 > URL: https://issues.apache.org/jira/browse/HDFS-13224 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13224.000.patch, HDFS-13224.001.patch, > HDFS-13224.002.patch, HDFS-13224.003.patch, HDFS-13224.004.patch, > HDFS-13224.005.patch, HDFS-13224.006.patch, HDFS-13224.007.patch, > HDFS-13224.008.patch, HDFS-13224.009.patch, HDFS-13224.010.patch > > > Currently, a mount point points to a single subcluster. We should be able to > spread files in a mount point across subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13219) RBF: Cluster information on Router is not correct when the Federation shares datanodes
[
https://issues.apache.org/jira/browse/HDFS-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13219:
--
Attachment: screenshot-2.png
> RBF: Cluster information on Router is not correct when the Federation shares
> datanodes
> --
>
> Key: HDFS-13219
> URL: https://issues.apache.org/jira/browse/HDFS-13219
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.9.0
>Reporter: Tao Jie
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> Now summary information on Router website aggregates summary of each
> nameservice. However in a typical federation cluster deployment, datanodes
> are shared among nameservices. Consider we have 2 namespaces and 100
> datanodes in one cluster. 100 datanodes are available for each namespace, but
> we see 200 datanodes on the router website. So does other information such as
> {{Total capacity}}, {{Remaining capacity}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13219) RBF: Cluster information on Router is not correct when the Federation shares datanodes
[
https://issues.apache.org/jira/browse/HDFS-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13219:
--
Attachment: screenshot-1.png
> RBF: Cluster information on Router is not correct when the Federation shares
> datanodes
> --
>
> Key: HDFS-13219
> URL: https://issues.apache.org/jira/browse/HDFS-13219
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.9.0
>Reporter: Tao Jie
>Priority: Major
> Attachments: screenshot-1.png
>
>
> Now summary information on Router website aggregates summary of each
> nameservice. However in a typical federation cluster deployment, datanodes
> are shared among nameservices. Consider we have 2 namespaces and 100
> datanodes in one cluster. 100 datanodes are available for each namespace, but
> we see 200 datanodes on the router website. So does other information such as
> {{Total capacity}}, {{Remaining capacity}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13258) Ozone: restructure Hdsl/Ozone code to separated maven subprojects
[ https://issues.apache.org/jira/browse/HDFS-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399685#comment-16399685 ] Anu Engineer commented on HDFS-13258: - [~elek] Thanks for the update. I will commit this as soon as we have a jenkins run. > Ozone: restructure Hdsl/Ozone code to separated maven subprojects > - > > Key: HDFS-13258 > URL: https://issues.apache.org/jira/browse/HDFS-13258 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HDFS-13258-HDFS-7240.001.patch, > HDFS-13258-HDFS-7240.002.patch, HDFS-13258-HDFS-7240.003.patch, > HDFS-13258-HDFS-7240.004.patch, HDFS-13258-HDFS-7240.005.patch, > HDFS-13258-HDFS-7240.006.patch, HDFS-13258-HDFS-7240.007.patch > > > According to the merge disucssion at hdfs-dev/hadoop-dev list it would be > easier to review/maintain hdsl/ozone code if it would be: > 1. separated from hadoop-hdfs/hadoop-common > 2. would be more structured > This jira is about moving out all the hdsl/ozone code to separated maven > projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13283) Percentage based Reserved Space Calculation for DataNode
[
https://issues.apache.org/jira/browse/HDFS-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399682#comment-16399682
]
Virajith Jalaparti commented on HDFS-13283:
---
Thanks for reporting this [~lukmajercak], and posting a patch.
One question -- why use an enum to define this and not directly specify the
class to use in the configuration? I think the latter makes this more general.
While {{PERCENTAGE}} seems like a simple alternative of the absolute value used
before, {{CONSERVATIVE}} and {{AGGRESSIVE}} seem like something specific
someone would want to configure. If someone wants to define a custom policy for
defining the reserved space at a later time, then they would have to add a new
enum. I would rather they just implement a new sub-class of
{{ReservedSpaceCalculator}} and directly configure. This will allow defining
and configuring the custom policies easier, and reduce divergence from trunk.
> Percentage based Reserved Space Calculation for DataNode
>
>
> Key: HDFS-13283
> URL: https://issues.apache.org/jira/browse/HDFS-13283
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Major
> Attachments: HDFS-13283.000.patch
>
>
> Currently, the only way to configure reserved disk space for non-HDFS data on
> a DataNode is a constant value via {{dfs.datanode.du.reserved}}. This can be
> an issue in non-heterogeneous clusters where size of DNs can differ. The
> proposed solution is to allow percentage based configuration (and their
> combination):
> # ABSOLUTE
> ** based on absolute number of reserved space
> # PERCENTAGE
> ** based on percentage of total capacity in the storage
> # CONSERVATIVE
> ** calculates both of the above and takes the one that will yield more
> reserved space
> # AGGRESSIVE
> ** calculates 1. 2. and takes the one that will yield less reserved space
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13287) TestINodeFile#testGetBlockType results in NPE when run alone
[
https://issues.apache.org/jira/browse/HDFS-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-13287:
--
Status: Patch Available (was: Open)
> TestINodeFile#testGetBlockType results in NPE when run alone
>
>
> Key: HDFS-13287
> URL: https://issues.apache.org/jira/browse/HDFS-13287
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Minor
> Attachments: HDFS-13287.01.patch
>
>
> When TestINodeFile#testGetBlockType is run by itself, it results in the
> following error:
> {code:java}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.218
> s <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.TestINodeFile
> [ERROR]
> testGetBlockType(org.apache.hadoop.hdfs.server.namenode.TestINodeFile) Time
> elapsed: 0.023 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager.getPolicyInfoByID(ErasureCodingPolicyManager.java:220)
> at
> org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager.getByID(ErasureCodingPolicyManager.java:208)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat.getBlockLayoutRedundancy(INodeFile.java:207)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.(INodeFile.java:266)
> at
> org.apache.hadoop.hdfs.server.namenode.TestINodeFile.createStripedINodeFile(TestINodeFile.java:112)
> at
> org.apache.hadoop.hdfs.server.namenode.TestINodeFile.testGetBlockType(TestINodeFile.java:299)
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13287) TestINodeFile#testGetBlockType results in NPE when run alone
[
https://issues.apache.org/jira/browse/HDFS-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399655#comment-16399655
]
Virajith Jalaparti commented on HDFS-13287:
---
The error results because {{ErasureCodingPolicyManager#getInstance}} is called
(creates a new instance of {{ErasureCodingPolicyManager}}) without a call to
{{ErasureCodingPolicyManager#init}} afterwards. The NPE doesn't occur when
{{TestINodeFile}} is run as a whole as one of the other tests initializes
{{ErasureCodingPolicyManager.instance}}.
Attached patch ( [^HDFS-13287.01.patch]) fixes this issue.
[~andrew.wang], [~ehiggs], can you take a look at this?
> TestINodeFile#testGetBlockType results in NPE when run alone
>
>
> Key: HDFS-13287
> URL: https://issues.apache.org/jira/browse/HDFS-13287
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Minor
> Attachments: HDFS-13287.01.patch
>
>
> When TestINodeFile#testGetBlockType is run by itself, it results in the
> following error:
> {code:java}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.218
> s <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.TestINodeFile
> [ERROR]
> testGetBlockType(org.apache.hadoop.hdfs.server.namenode.TestINodeFile) Time
> elapsed: 0.023 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager.getPolicyInfoByID(ErasureCodingPolicyManager.java:220)
> at
> org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager.getByID(ErasureCodingPolicyManager.java:208)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat.getBlockLayoutRedundancy(INodeFile.java:207)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.(INodeFile.java:266)
> at
> org.apache.hadoop.hdfs.server.namenode.TestINodeFile.createStripedINodeFile(TestINodeFile.java:112)
> at
> org.apache.hadoop.hdfs.server.namenode.TestINodeFile.testGetBlockType(TestINodeFile.java:299)
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13287) TestINodeFile#testGetBlockType results in NPE when run alone
[
https://issues.apache.org/jira/browse/HDFS-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-13287:
--
Attachment: HDFS-13287.01.patch
> TestINodeFile#testGetBlockType results in NPE when run alone
>
>
> Key: HDFS-13287
> URL: https://issues.apache.org/jira/browse/HDFS-13287
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Minor
> Attachments: HDFS-13287.01.patch
>
>
> When TestINodeFile#testGetBlockType is run by itself, it results in the
> following error:
> {code:java}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.218
> s <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.TestINodeFile
> [ERROR]
> testGetBlockType(org.apache.hadoop.hdfs.server.namenode.TestINodeFile) Time
> elapsed: 0.023 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager.getPolicyInfoByID(ErasureCodingPolicyManager.java:220)
> at
> org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager.getByID(ErasureCodingPolicyManager.java:208)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat.getBlockLayoutRedundancy(INodeFile.java:207)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.(INodeFile.java:266)
> at
> org.apache.hadoop.hdfs.server.namenode.TestINodeFile.createStripedINodeFile(TestINodeFile.java:112)
> at
> org.apache.hadoop.hdfs.server.namenode.TestINodeFile.testGetBlockType(TestINodeFile.java:299)
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13283) Percentage based Reserved Space Calculation for DataNode
[
https://issues.apache.org/jira/browse/HDFS-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399650#comment-16399650
]
genericqa commented on HDFS-13283:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 53s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 22 new + 449 unchanged - 0 fixed = 471 total (was 449) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 12s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}157m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-13283 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12914586/HDFS-13283.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux c7bf7b0c82b1 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 252c2b4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/23489/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
[jira] [Created] (HDFS-13287) TestINodeFile#testGetBlockType results in NPE when run alone
Virajith Jalaparti created HDFS-13287:
-
Summary: TestINodeFile#testGetBlockType results in NPE when run
alone
Key: HDFS-13287
URL: https://issues.apache.org/jira/browse/HDFS-13287
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Virajith Jalaparti
Assignee: Virajith Jalaparti
When TestINodeFile#testGetBlockType is run by itself, it results in the
following error:
{code:java}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.218 s
<<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.TestINodeFile
[ERROR] testGetBlockType(org.apache.hadoop.hdfs.server.namenode.TestINodeFile)
Time elapsed: 0.023 s <<< ERROR!
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager.getPolicyInfoByID(ErasureCodingPolicyManager.java:220)
at
org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager.getByID(ErasureCodingPolicyManager.java:208)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat.getBlockLayoutRedundancy(INodeFile.java:207)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.(INodeFile.java:266)
at
org.apache.hadoop.hdfs.server.namenode.TestINodeFile.createStripedINodeFile(TestINodeFile.java:112)
at
org.apache.hadoop.hdfs.server.namenode.TestINodeFile.testGetBlockType(TestINodeFile.java:299)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13224) RBF: Resolvers to support mount points across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-13224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399642#comment-16399642
]
genericqa commented on HDFS-13224:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 41s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}128m 26s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}181m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits |
| | hadoop.hdfs.server.namenode.TestMetaSave |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-13224 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12914581/HDFS-13224.010.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| uname | Linux 6dc5e54e174d 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 252c2b4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23484/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
|
[jira] [Created] (HDFS-13286) Add haadmin commands to transition between standby and observer
Chao Sun created HDFS-13286:
---
Summary: Add haadmin commands to transition between standby and
observer
Key: HDFS-13286
URL: https://issues.apache.org/jira/browse/HDFS-13286
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Chao Sun
Assignee: Chao Sun
As discussed in HDFS-12975, we should allow explicit transition between standby
and observer through haadmin command, such as:
{code}
haadmin -transitionToObserver
{code}
Initially we should support transition from observer to standby, and standby to
observer.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13280) WebHDFS: Fix NPE in get snasphottable directory list call
[
https://issues.apache.org/jira/browse/HDFS-13280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399628#comment-16399628
]
genericqa commented on HDFS-13280:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 12s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 47s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}122m 7s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}169m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
|
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-13280 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12914563/HDFS-13280.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 09687f30e266 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12
21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 252c2b4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23485/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23485/testReport/ |
| Max. process+thread count | 3542 (vs. ulimit of 1) |
| modules |
[jira] [Commented] (HDFS-12422) Replace DataNode in Pipeline when waiting for Last Packet fails
[
https://issues.apache.org/jira/browse/HDFS-12422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399612#comment-16399612
]
genericqa commented on HDFS-12422:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 52s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
22s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}100m 45s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-12422 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12914569/HDFS-12422.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 24e3bcb7fb6e 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12
21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HDFS-12723) TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing consistently.
[
https://issues.apache.org/jira/browse/HDFS-12723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399606#comment-16399606
]
Ajay Kumar commented on HDFS-12723:
---
[~elgoiri], Created [HDFS-13285] to address that. I think it applies to only
3.X. Thanks for review.
> TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing
> consistently.
>
>
> Key: HDFS-12723
> URL: https://issues.apache.org/jira/browse/HDFS-12723
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Rushabh S Shah
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-12723.000.patch, HDFS-12723.001.patch,
> HDFS-12723.002.patch
>
>
> TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks is timing
> out consistently on my local machine.
> {noformat}
> Running org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 132.405 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
> testReadFileWithMissingBlocks(org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks)
> Time elapsed: 132.171 sec <<< ERROR!
> java.util.concurrent.TimeoutException: Timed out waiting for /foo to have all
> the internalBlocks
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:295)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:256)
> at
> org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.readFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:98)
> at
> org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:82)
> Results :
> Tests in error:
>
> TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks:82->readFileWithMissingBlocks:98
> » Timeout
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13285) Improve runtime for TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks
Ajay Kumar created HDFS-13285: - Summary: Improve runtime for TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks Key: HDFS-13285 URL: https://issues.apache.org/jira/browse/HDFS-13285 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ajay Kumar TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks takes anywhere b/w 2-4 minutes depending on host machine. Jira intends to make it leaner. cc: [~elgoiri] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated HDFS-13215: --- Attachment: HDFS-13215.003.patch > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13284) Adjust criteria for LowRedundancyBlocks.QUEUE_VERY_LOW_REDUNDANCY
[ https://issues.apache.org/jira/browse/HDFS-13284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399574#comment-16399574 ] Íñigo Goiri commented on HDFS-13284: [~lukmajercak], could you give a high levle description? This code is pretty old. [~ste...@apache.org], could you take a look? > Adjust criteria for LowRedundancyBlocks.QUEUE_VERY_LOW_REDUNDANCY > - > > Key: HDFS-13284 > URL: https://issues.apache.org/jira/browse/HDFS-13284 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, namenode >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HDFS-13284.000.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12723) TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing consistently.
[
https://issues.apache.org/jira/browse/HDFS-12723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399570#comment-16399570
]
Íñigo Goiri commented on HDFS-12723:
+1 on [^HDFS-12723.002.patch].
Let's wait for Yetus to double check though.
[~ajayydv], do you mind opening the JIRA to discuss how to make this test
lighter?
Which versions does this apply? Only 3.X, AFAIK, striped is not available in
2.X, right?
> TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing
> consistently.
>
>
> Key: HDFS-12723
> URL: https://issues.apache.org/jira/browse/HDFS-12723
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Rushabh S Shah
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-12723.000.patch, HDFS-12723.001.patch,
> HDFS-12723.002.patch
>
>
> TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks is timing
> out consistently on my local machine.
> {noformat}
> Running org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 132.405 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
> testReadFileWithMissingBlocks(org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks)
> Time elapsed: 132.171 sec <<< ERROR!
> java.util.concurrent.TimeoutException: Timed out waiting for /foo to have all
> the internalBlocks
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:295)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:256)
> at
> org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.readFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:98)
> at
> org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:82)
> Results :
> Tests in error:
>
> TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks:82->readFileWithMissingBlocks:98
> » Timeout
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13284) Adjust criteria for LowRedundancyBlocks.QUEUE_VERY_LOW_REDUNDANCY
[ https://issues.apache.org/jira/browse/HDFS-13284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HDFS-13284: -- Attachment: HDFS-13284.000.patch > Adjust criteria for LowRedundancyBlocks.QUEUE_VERY_LOW_REDUNDANCY > - > > Key: HDFS-13284 > URL: https://issues.apache.org/jira/browse/HDFS-13284 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, namenode >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HDFS-13284.000.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated HDFS-13215: --- Attachment: HDFS-13215.002.patch > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13284) Adjust criteria for LowRedundancyBlocks.QUEUE_VERY_LOW_REDUNDANCY
Lukas Majercak created HDFS-13284: - Summary: Adjust criteria for LowRedundancyBlocks.QUEUE_VERY_LOW_REDUNDANCY Key: HDFS-13284 URL: https://issues.apache.org/jira/browse/HDFS-13284 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs, namenode Reporter: Lukas Majercak Assignee: Lukas Majercak -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12723) TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing consistently.
[
https://issues.apache.org/jira/browse/HDFS-12723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399556#comment-16399556
]
Ajay Kumar commented on HDFS-12723:
---
[~elgoiri] thanks for testing this, updated patch v2 has increased DNs.
> TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing
> consistently.
>
>
> Key: HDFS-12723
> URL: https://issues.apache.org/jira/browse/HDFS-12723
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Rushabh S Shah
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-12723.000.patch, HDFS-12723.001.patch,
> HDFS-12723.002.patch
>
>
> TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks is timing
> out consistently on my local machine.
> {noformat}
> Running org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 132.405 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
> testReadFileWithMissingBlocks(org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks)
> Time elapsed: 132.171 sec <<< ERROR!
> java.util.concurrent.TimeoutException: Timed out waiting for /foo to have all
> the internalBlocks
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:295)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:256)
> at
> org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.readFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:98)
> at
> org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:82)
> Results :
> Tests in error:
>
> TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks:82->readFileWithMissingBlocks:98
> » Timeout
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12723) TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing consistently.
[
https://issues.apache.org/jira/browse/HDFS-12723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399534#comment-16399534
]
Ajay Kumar edited comment on HDFS-12723 at 3/14/18 10:41 PM:
-
[~elgoiri], this timeout is different than test timeout, this indicates no of
total live datanodes are less than required no 9. I was thinking of increasing
numDNs to little more than 9 as even 1 dead/slow DN will result in test
failure. Could you please try again by {{private final int numDNs = dataBlocks
+ parityBlocks;}} set to {{private final int numDNs = dataBlocks +
parityBlocks+2;}}
If you agree with increasing {{numDNs}} i can update the patch.
was (Author: ajayydv):
[~elgoiri], this timeout is different than test timeout, this indicates no of
total live datanodes are less than required no 9. I was thinking of increasing
numDNs to little more than 9 as even 1 dead/slow DN will result in test
failure. Could you please try again by {{private final int numDNs = dataBlocks
+ parityBlocks;}} set to {{private final int numDNs = dataBlocks +
parityBlocks+2;}}
If you agree i with increasing {{numDNs}} i can update the patch.
> TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing
> consistently.
>
>
> Key: HDFS-12723
> URL: https://issues.apache.org/jira/browse/HDFS-12723
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Rushabh S Shah
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-12723.000.patch, HDFS-12723.001.patch,
> HDFS-12723.002.patch
>
>
> TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks is timing
> out consistently on my local machine.
> {noformat}
> Running org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 132.405 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
> testReadFileWithMissingBlocks(org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks)
> Time elapsed: 132.171 sec <<< ERROR!
> java.util.concurrent.TimeoutException: Timed out waiting for /foo to have all
> the internalBlocks
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:295)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:256)
> at
> org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.readFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:98)
> at
> org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:82)
> Results :
> Tests in error:
>
> TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks:82->readFileWithMissingBlocks:98
> » Timeout
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12723) TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing consistently.
[
https://issues.apache.org/jira/browse/HDFS-12723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HDFS-12723:
--
Attachment: HDFS-12723.002.patch
> TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing
> consistently.
>
>
> Key: HDFS-12723
> URL: https://issues.apache.org/jira/browse/HDFS-12723
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Rushabh S Shah
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-12723.000.patch, HDFS-12723.001.patch,
> HDFS-12723.002.patch
>
>
> TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks is timing
> out consistently on my local machine.
> {noformat}
> Running org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 132.405 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
> testReadFileWithMissingBlocks(org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks)
> Time elapsed: 132.171 sec <<< ERROR!
> java.util.concurrent.TimeoutException: Timed out waiting for /foo to have all
> the internalBlocks
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:295)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:256)
> at
> org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.readFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:98)
> at
> org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:82)
> Results :
> Tests in error:
>
> TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks:82->readFileWithMissingBlocks:98
> » Timeout
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12723) TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing consistently.
[
https://issues.apache.org/jira/browse/HDFS-12723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399548#comment-16399548
]
Íñigo Goiri commented on HDFS-12723:
Thanks [~ajayydv], with two extra DNs, the unit test passes.
Let's get this committed with the +2 DNs to fix this test.
We should open a new JIRA to make this test leaner; right now, in my machine it
takes around 1.5GB and 2 full cores for 160 seconds.
> TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing
> consistently.
>
>
> Key: HDFS-12723
> URL: https://issues.apache.org/jira/browse/HDFS-12723
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Rushabh S Shah
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-12723.000.patch, HDFS-12723.001.patch
>
>
> TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks is timing
> out consistently on my local machine.
> {noformat}
> Running org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 132.405 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
> testReadFileWithMissingBlocks(org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks)
> Time elapsed: 132.171 sec <<< ERROR!
> java.util.concurrent.TimeoutException: Timed out waiting for /foo to have all
> the internalBlocks
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:295)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:256)
> at
> org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.readFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:98)
> at
> org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:82)
> Results :
> Tests in error:
>
> TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks:82->readFileWithMissingBlocks:98
> » Timeout
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13280) WebHDFS: Fix NPE in get snasphottable directory list call
[ https://issues.apache.org/jira/browse/HDFS-13280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13280: -- Target Version/s: 3.1.0 (was: 3.1.0, 3.0.2) > WebHDFS: Fix NPE in get snasphottable directory list call > - > > Key: HDFS-13280 > URL: https://issues.apache.org/jira/browse/HDFS-13280 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13280.001.patch, HDFS-13280.002.patch, > HDFS-13280.003.patch > > > WebHdfs throws NPE when snapshottable directory status list is null. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13258) Ozone: restructure Hdsl/Ozone code to separated maven subprojects
[ https://issues.apache.org/jira/browse/HDFS-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399540#comment-16399540 ] Elek, Marton commented on HDFS-13258: - version 7 contains a small fix: OzoneMultiCluster should configure the requried hdsl/ozone datanode services by default. > Ozone: restructure Hdsl/Ozone code to separated maven subprojects > - > > Key: HDFS-13258 > URL: https://issues.apache.org/jira/browse/HDFS-13258 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HDFS-13258-HDFS-7240.001.patch, > HDFS-13258-HDFS-7240.002.patch, HDFS-13258-HDFS-7240.003.patch, > HDFS-13258-HDFS-7240.004.patch, HDFS-13258-HDFS-7240.005.patch, > HDFS-13258-HDFS-7240.006.patch, HDFS-13258-HDFS-7240.007.patch > > > According to the merge disucssion at hdfs-dev/hadoop-dev list it would be > easier to review/maintain hdsl/ozone code if it would be: > 1. separated from hadoop-hdfs/hadoop-common > 2. would be more structured > This jira is about moving out all the hdsl/ozone code to separated maven > projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13258) Ozone: restructure Hdsl/Ozone code to separated maven subprojects
[ https://issues.apache.org/jira/browse/HDFS-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDFS-13258: Attachment: HDFS-13258-HDFS-7240.007.patch > Ozone: restructure Hdsl/Ozone code to separated maven subprojects > - > > Key: HDFS-13258 > URL: https://issues.apache.org/jira/browse/HDFS-13258 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HDFS-13258-HDFS-7240.001.patch, > HDFS-13258-HDFS-7240.002.patch, HDFS-13258-HDFS-7240.003.patch, > HDFS-13258-HDFS-7240.004.patch, HDFS-13258-HDFS-7240.005.patch, > HDFS-13258-HDFS-7240.006.patch, HDFS-13258-HDFS-7240.007.patch > > > According to the merge disucssion at hdfs-dev/hadoop-dev list it would be > easier to review/maintain hdsl/ozone code if it would be: > 1. separated from hadoop-hdfs/hadoop-common > 2. would be more structured > This jira is about moving out all the hdsl/ozone code to separated maven > projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12723) TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing consistently.
[
https://issues.apache.org/jira/browse/HDFS-12723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399534#comment-16399534
]
Ajay Kumar commented on HDFS-12723:
---
[~elgoiri], this timeout is different than test timeout, this indicates no of
total live datanodes are less than required no 9. I was thinking of increasing
numDNs to little more than 9 as even 1 dead/slow DN will result in test
failure. Could you please try again by {{private final int numDNs = dataBlocks
+ parityBlocks;}} set to {{private final int numDNs = dataBlocks +
parityBlocks+2;}}
If you agree i with increasing {{numDNs}} i can update the patch.
> TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing
> consistently.
>
>
> Key: HDFS-12723
> URL: https://issues.apache.org/jira/browse/HDFS-12723
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Rushabh S Shah
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-12723.000.patch, HDFS-12723.001.patch
>
>
> TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks is timing
> out consistently on my local machine.
> {noformat}
> Running org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 132.405 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
> testReadFileWithMissingBlocks(org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks)
> Time elapsed: 132.171 sec <<< ERROR!
> java.util.concurrent.TimeoutException: Timed out waiting for /foo to have all
> the internalBlocks
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:295)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:256)
> at
> org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.readFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:98)
> at
> org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:82)
> Results :
> Tests in error:
>
> TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks:82->readFileWithMissingBlocks:98
> » Timeout
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13198) RBF: RouterHeartbeatService throws out CachedStateStore related exceptions when starting router
[
https://issues.apache.org/jira/browse/HDFS-13198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399497#comment-16399497
]
Wei Yan commented on HDFS-13198:
[~elgoiri] thanks for the review.
> RBF: RouterHeartbeatService throws out CachedStateStore related exceptions
> when starting router
> ---
>
> Key: HDFS-13198
> URL: https://issues.apache.org/jira/browse/HDFS-13198
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.2, 3.2.0
>
> Attachments: HDFS-13198.000.patch, HDFS-13198.001.patch,
> HDFS-13198.002.patch, HDFS-13198.003.patch, HDFS-13198.004.patch,
> HDFS-13198.005.patch
>
>
> Exception looks like:
> {code:java}
> 2018-02-23 19:04:56,341 ERROR router.RouterHeartbeatService: Cannot get
> version for class
> org.apache.hadoop.hdfs.server.federation.store.MembershipStore: Cached State
> Store not initialized, MembershipState records not valid
> 2018-02-23 19:04:56,341 ERROR router.RouterHeartbeatService: Cannot get
> version for class
> org.apache.hadoop.hdfs.server.federation.store.MountTableStore: Cached State
> Store not initialized, MountTable records not valid
> Exception in thread "Router Heartbeat Async" java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreSerializableImpl.serialize(StateStoreSerializableImpl.java:60)
> at
> org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreZooKeeperImpl.putAll(StateStoreZooKeeperImpl.java:191)
> at
> org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreBaseImpl.put(StateStoreBaseImpl.java:75)
> at
> org.apache.hadoop.hdfs.server.federation.store.impl.RouterStoreImpl.routerHeartbeat(RouterStoreImpl.java:88)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterHeartbeatService.updateStateStore(RouterHeartbeatService.java:95)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterHeartbeatService.access$000(RouterHeartbeatService.java:43)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterHeartbeatService$1.run(RouterHeartbeatService.java:68)
> at java.lang.Thread.run(Thread.java:748){code}
> This is because, during starting the Router, the CachedStateStore hasn't been
> initialized and cannot serve requests. Although the router will still be
> started, it would be better to fix the exceptions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13283) Percentage based Reserved Space Calculation for DataNode
[
https://issues.apache.org/jira/browse/HDFS-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399495#comment-16399495
]
Íñigo Goiri commented on HDFS-13283:
I think the documentation in hdfs-default.xml should be good enough.
If anybody has a suggestion whether we can add documentation for this.
> Percentage based Reserved Space Calculation for DataNode
>
>
> Key: HDFS-13283
> URL: https://issues.apache.org/jira/browse/HDFS-13283
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Major
> Attachments: HDFS-13283.000.patch
>
>
> Currently, the only way to configure reserved disk space for non-HDFS data on
> a DataNode is a constant value via {{dfs.datanode.du.reserved}}. This can be
> an issue in non-heterogeneous clusters where size of DNs can differ. The
> proposed solution is to allow percentage based configuration (and their
> combination):
> # ABSOLUTE
> ** based on absolute number of reserved space
> # PERCENTAGE
> ** based on percentage of total capacity in the storage
> # CONSERVATIVE
> ** calculates both of the above and takes the one that will yield more
> reserved space
> # AGGRESSIVE
> ** calculates 1. 2. and takes the one that will yield less reserved space
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13276) TestDistributedFileSystem doesn't cleanup MiniDFSCluster if test times out
[ https://issues.apache.org/jira/browse/HDFS-13276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399492#comment-16399492 ] Íñigo Goiri commented on HDFS-13276: The output from Yetus for [^HDFS-13276.002.patch] is not there. It says is failing TestDistributedFilesystem, [~lukmajercak], do you mind rerunning this? > TestDistributedFileSystem doesn't cleanup MiniDFSCluster if test times out > -- > > Key: HDFS-13276 > URL: https://issues.apache.org/jira/browse/HDFS-13276 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Íñigo Goiri >Assignee: Lukas Majercak >Priority: Major > Attachments: HDFS-13276-branch-2.000.patch, HDFS-13276.000.patch, > HDFS-13276.001.patch, HDFS-13276.002.patch > > > If a unit tests times out, it may leave a MiniDFSCluster behing. This is > particularly bad in Windows where the new MiniDFSCluster cannot start and all > tests will fail after this one. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13283) Percentage based Reserved Space Calculation for DataNode
[
https://issues.apache.org/jira/browse/HDFS-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399490#comment-16399490
]
Lukas Majercak commented on HDFS-13283:
---
We've been running this in production for 4months~ with the following
configurations :
dfs.datanode.du.reserved.type=CONSERVATIVE
dfs.datanode.du.reserved.pct=10
dfs.datanode.du.reserved=107374182400
This allows us to leave at least 100GB reserved space and at the same time
adjust the reserved space to the DNs disk size so that we don't use more than
90%.
> Percentage based Reserved Space Calculation for DataNode
>
>
> Key: HDFS-13283
> URL: https://issues.apache.org/jira/browse/HDFS-13283
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Major
> Attachments: HDFS-13283.000.patch
>
>
> Currently, the only way to configure reserved disk space for non-HDFS data on
> a DataNode is a constant value via {{dfs.datanode.du.reserved}}. This can be
> an issue in non-heterogeneous clusters where size of DNs can differ. The
> proposed solution is to allow percentage based configuration (and their
> combination):
> # ABSOLUTE
> ** based on absolute number of reserved space
> # PERCENTAGE
> ** based on percentage of total capacity in the storage
> # CONSERVATIVE
> ** calculates both of the above and takes the one that will yield more
> reserved space
> # AGGRESSIVE
> ** calculates 1. 2. and takes the one that will yield less reserved space
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13198) RBF: RouterHeartbeatService throws out CachedStateStore related exceptions when starting router
[
https://issues.apache.org/jira/browse/HDFS-13198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399458#comment-16399458
]
Hudson commented on HDFS-13198:
---
ABORTED: Integrated in Jenkins build Hadoop-trunk-Commit #13838 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13838/])
HDFS-13198. RBF: RouterHeartbeatService throws out CachedStateStore (inigoiri:
rev 41fc7f80beb32f4a1fac73faf7c9116ad5ee3420)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterHeartbeatService.java
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterHeartbeatService.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/driver/impl/StateStoreZooKeeperImpl.java
> RBF: RouterHeartbeatService throws out CachedStateStore related exceptions
> when starting router
> ---
>
> Key: HDFS-13198
> URL: https://issues.apache.org/jira/browse/HDFS-13198
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.2, 3.2.0
>
> Attachments: HDFS-13198.000.patch, HDFS-13198.001.patch,
> HDFS-13198.002.patch, HDFS-13198.003.patch, HDFS-13198.004.patch,
> HDFS-13198.005.patch
>
>
> Exception looks like:
> {code:java}
> 2018-02-23 19:04:56,341 ERROR router.RouterHeartbeatService: Cannot get
> version for class
> org.apache.hadoop.hdfs.server.federation.store.MembershipStore: Cached State
> Store not initialized, MembershipState records not valid
> 2018-02-23 19:04:56,341 ERROR router.RouterHeartbeatService: Cannot get
> version for class
> org.apache.hadoop.hdfs.server.federation.store.MountTableStore: Cached State
> Store not initialized, MountTable records not valid
> Exception in thread "Router Heartbeat Async" java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreSerializableImpl.serialize(StateStoreSerializableImpl.java:60)
> at
> org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreZooKeeperImpl.putAll(StateStoreZooKeeperImpl.java:191)
> at
> org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreBaseImpl.put(StateStoreBaseImpl.java:75)
> at
> org.apache.hadoop.hdfs.server.federation.store.impl.RouterStoreImpl.routerHeartbeat(RouterStoreImpl.java:88)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterHeartbeatService.updateStateStore(RouterHeartbeatService.java:95)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterHeartbeatService.access$000(RouterHeartbeatService.java:43)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterHeartbeatService$1.run(RouterHeartbeatService.java:68)
> at java.lang.Thread.run(Thread.java:748){code}
> This is because, during starting the Router, the CachedStateStore hasn't been
> initialized and cannot serve requests. Although the router will still be
> started, it would be better to fix the exceptions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13283) Percentage based Reserved Space Calculation for DataNode
[
https://issues.apache.org/jira/browse/HDFS-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399444#comment-16399444
]
Íñigo Goiri commented on HDFS-13283:
Thanks [~lukmajercak] for [^HDFS-13283.000.patch].
The unit tests are a pretty good guide for the use of this feature.
Can you share what's the experience with this?
> Percentage based Reserved Space Calculation for DataNode
>
>
> Key: HDFS-13283
> URL: https://issues.apache.org/jira/browse/HDFS-13283
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Major
> Attachments: HDFS-13283.000.patch
>
>
> Currently, the only way to configure reserved disk space for non-HDFS data on
> a DataNode is a constant value via {{dfs.datanode.du.reserved}}. This can be
> an issue in non-heterogeneous clusters where size of DNs can differ. The
> proposed solution is to allow percentage based configuration (and their
> combination):
> # ABSOLUTE
> ** based on absolute number of reserved space
> # PERCENTAGE
> ** based on percentage of total capacity in the storage
> # CONSERVATIVE
> ** calculates both of the above and takes the one that will yield more
> reserved space
> # AGGRESSIVE
> ** calculates 1. 2. and takes the one that will yield less reserved space
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
