[jira] [Created] (HDFS-11685) TestDistributedFileSystem.java fails to compile

[John Zhuge (JIRA)]

John Zhuge created HDFS-11685:
-

 Summary: TestDistributedFileSystem.java fails to compile
 Key: HDFS-11685
 URL: https://issues.apache.org/jira/browse/HDFS-11685
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 2.9.0
Reporter: John Zhuge
Assignee: John Zhuge


{noformat}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile 
(default-testCompile) on project hadoop-hdfs: Compilation failure: Compilation 
failure:
[ERROR] 
/Users/jzhuge/hadoop-commit/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java:[1244,28]
 instance has private access in 
org.apache.hadoop.hdfs.server.datanode.DataNodeFaultInjector
[ERROR] 
/Users/jzhuge/hadoop-commit/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java:[1254,28]
 instance has private access in 
org.apache.hadoop.hdfs.server.datanode.DataNodeFaultInjector
{noformat}

{{DataNodeFaultInjector#instance}} was changed to private by HDFS-11660,



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11684) Potential scalability bug on datanode recommission

[Riza Suminto (JIRA)]

Riza Suminto created HDFS-11684:
---

 Summary: Potential scalability bug on datanode recommission
 Key: HDFS-11684
 URL: https://issues.apache.org/jira/browse/HDFS-11684
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.7.3
Reporter: Riza Suminto


We are academic researcher working on scalability bug detection.

Upon analyzing Hadoop code, our static analysis program detect
potential scalability problem along this function calls:

DatanodeManager.refreshNodes() -> DatanodeManager.refreshDatanodes() -> 
DecommissionManager.stopDecommission() -> 
BlockManager.processExtraRedundancyBlocksOnInService()

DatanodeManager.refreshNodes() call namesystem.writeLock(),
which will lock FSNamesystem. While locking FSNamesystem, it does
nested loop over datanodes in DatanodeManager.refreshDatanodes() and
over datanode blocks in BlockManager.processExtraRedundancyBlocksOnInService().

Looking only at number of executed lines, refreshDatanodes will
execute N*(136*B+137) lines of codes in total, where N is number of
datanodes and B is number of blocks per node.

This is a potential scalability bug that may render FSNamesystem
unresponsive for long time if refreshNodes trigger recommissioning of
many fat datanodes, having large number of over replicated datablocks.

This bug also discussed in HDFS-10477, but still left unresolved.

In older hadoop version, our program also detect similar scalability
problem in decommissioning datanodes, which has been fixed on
HADOOP-4061. However, recommissioning still have scalabillity problem
in current version.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-11683) getErasureCodingPolicyByName should throw a checked exception

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang resolved HDFS-11683.

Resolution: Not A Problem

Looking more, it seems like this is actually pretty common in our API. NVM. 

> getErasureCodingPolicyByName should throw a checked exception
> -
>
> Key: HDFS-11683
> URL: https://issues.apache.org/jira/browse/HDFS-11683
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Andrew Wang
>  Labels: newbie
>
> getErasureCodingPolicyByName throws a HadoopIllegalArgumentException, which 
> is a RuntimeException. This is basically only used by CLI tools and client 
> code, we should throw a different exception in the NameNode.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11683) getErasureCodingPolicyByName should throw a checked exception

[Andrew Wang (JIRA)]

Andrew Wang created HDFS-11683:
--

 Summary: getErasureCodingPolicyByName should throw a checked 
exception
 Key: HDFS-11683
 URL: https://issues.apache.org/jira/browse/HDFS-11683
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: erasure-coding
Affects Versions: 3.0.0-alpha3
Reporter: Andrew Wang


getErasureCodingPolicyByName throws a HadoopIllegalArgumentException, which is 
a RuntimeException. This is basically only used by CLI tools and client code, 
we should throw a different exception in the NameNode.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/381/

[Apr 18, 2017 7:43:50 AM] (aajisaka) HADOOP-14318. Remove non-existent setfattr 
command option from
[Apr 18, 2017 8:14:22 AM] (aajisaka) HADOOP-14315. Python example in the rack 
awareness document doesn't work
[Apr 18, 2017 4:28:52 PM] (jzhuge) HDFS-11665. 
HttpFSServerWebServer$deprecateEnv may leak secret.
[Apr 18, 2017 4:29:16 PM] (jzhuge) HADOOP-14317. KMSWebServer$deprecateEnv may 
leak secret. Contributed by
[Apr 18, 2017 5:29:44 PM] (cdouglas) YARN-6451. Add RM monitor validating 
metrics invariants. Contributed by
[Apr 19, 2017 4:11:07 AM] (jzhuge) HDFS-11531. Expose hedged read metrics via 
libHDFS API. Contributed by




-1 overall


The following subsystems voted -1:
asflicense unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.server.datanode.TestDataNodeMXBean 
   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport 
   hadoop.hdfs.TestDistributedFileSystem 
   hadoop.hdfs.TestPersistBlocks 
   hadoop.hdfs.server.datanode.TestDataNodeUUID 
   hadoop.yarn.server.resourcemanager.TestRMRestart 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestDiskFailures 
   hadoop.mapred.TestJobEndNotifier 
   hadoop.mapred.TestMRTimelineEventHandling 
   hadoop.tools.TestDistCpSystem 
   hadoop.tools.TestHadoopArchiveLogsRunner 
   hadoop.metrics2.impl.TestKafkaMetrics 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/381/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/381/artifact/out/diff-compile-javac-root.txt
  [184K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/381/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/381/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/381/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/381/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/381/artifact/out/whitespace-eol.txt
  [12M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/381/artifact/out/whitespace-tabs.txt
  [1.2M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/381/artifact/out/diff-javadoc-javadoc-root.txt
  [2.2M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/381/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [452K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/381/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [60K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/381/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [324K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/381/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
  [56K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/381/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
  [88K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/381/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/381/artifact/out/patch-unit-hadoop-tools_hadoop-archive-logs.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/381/artifact/out/patch-unit-hadoop-tools_hadoop-kafka.txt
  [8.0K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/381/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11682) TestBalancer#testBalancerWithStripedFile is flaky

[Andrew Wang (JIRA)]

Andrew Wang created HDFS-11682:
--

 Summary: TestBalancer#testBalancerWithStripedFile is flaky
 Key: HDFS-11682
 URL: https://issues.apache.org/jira/browse/HDFS-11682
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 3.0.0-alpha3
Reporter: Andrew Wang


Saw this fail in two different ways on a precommit run, but pass locally.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/293/

[Apr 18, 2017 4:28:52 PM] (jzhuge) HDFS-11665. 
HttpFSServerWebServer$deprecateEnv may leak secret.
[Apr 18, 2017 4:29:16 PM] (jzhuge) HADOOP-14317. KMSWebServer$deprecateEnv may 
leak secret. Contributed by
[Apr 18, 2017 5:29:44 PM] (cdouglas) YARN-6451. Add RM monitor validating 
metrics invariants. Contributed by
[Apr 19, 2017 4:11:07 AM] (jzhuge) HDFS-11531. Expose hedged read metrics via 
libHDFS API. Contributed by
[Apr 19, 2017 9:21:44 AM] (stevel) HADOOP-14321. explicitly exclude s3a root 
dir ITests from parallel runs.




-1 overall


The following subsystems voted -1:
compile mvninstall unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery 
   hadoop.hdfs.TestEncryptedTransfer 
   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer 
   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 
   hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService 
   hadoop.mapred.TestShuffleHandler 
   hadoop.tools.TestHadoopArchiveLogsRunner 
   hadoop.metrics2.impl.TestKafkaMetrics 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.yarn.server.timeline.TestRollingLevelDB 
   hadoop.yarn.server.timeline.TestTimelineDataManager 
   hadoop.yarn.server.timeline.TestLeveldbTimelineStore 
   hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore 
   hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore 
   
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer 
   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer 
   hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore 
   hadoop.yarn.server.resourcemanager.TestRMRestart 
   
hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore 
   hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient 
   hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore 

Timed out junit tests :

   org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache 
  

   mvninstall:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/293/artifact/out/patch-mvninstall-root.txt
  [492K]

   compile:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/293/artifact/out/patch-compile-root.txt
  [20K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/293/artifact/out/patch-compile-root.txt
  [20K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/293/artifact/out/patch-compile-root.txt
  [20K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/293/artifact/out/patch-unit-hadoop-assemblies.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/293/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [280K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/293/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/293/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt
  [52K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/293/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/293/artifact/out/patch-unit-hadoop-tools_hadoop-archive-logs.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/293/artifact/out/patch-unit-hadoop-tools_hadoop-kafka.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/293/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/293/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [52K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/293/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [16K]
  

Re: HADOOP-14316: Switching from Findbugs to Spotbugs

[Allen Wittenauer]

> On Apr 19, 2017, at 10:52 AM, Wei-Chiu Chuang  wrote:
> That sounds scary. Would you mind to share the list of bugs that spotbugs 
> found? Sounds like some of them may warrant new blockers jiras for Hadoop 3.


I've added the list to the JIRA.
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11681) DatanodeStorageInfo#getBlockIterator() should return an iterator to an unmodifiable set.

[Virajith Jalaparti (JIRA)]

Virajith Jalaparti created HDFS-11681:
-

 Summary: DatanodeStorageInfo#getBlockIterator() should return an 
iterator to an unmodifiable set.
 Key: HDFS-11681
 URL: https://issues.apache.org/jira/browse/HDFS-11681
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Virajith Jalaparti


The iterator from {{DatanodeStorageInfo#getBlockIterator()}} should not be 
modifiable. Otherwise, calling {{remove}} on the iterator will remove blocks 
from {{DatanodeStorageInfo.blocks}}, a function that has to be performed by 
calling {{DatanodeStorageInfo#removeBlock}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: HADOOP-14316: Switching from Findbugs to Spotbugs

[Wei-Chiu Chuang]

Hi Allen,
That sounds scary. Would you mind to share the list of bugs that spotbugs 
found? Sounds like some of them may warrant new blockers jiras for Hadoop 3.

Thank you,
Wei-Chiu Chuang

> On Apr 19, 2017, at 10:44 AM, Allen Wittenauer  
> wrote:
> 
> HADOOP-14316



signature.asc
Description: Message signed with OpenPGP using GPGMail

[jira] [Created] (HDFS-11678) Ozone: SCM CLI: Implement get container metrics command

[Weiwei Yang (JIRA)]

Weiwei Yang created HDFS-11678:
--

 Summary: Ozone: SCM CLI: Implement get container metrics command
 Key: HDFS-11678
 URL: https://issues.apache.org/jira/browse/HDFS-11678
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Weiwei Yang


Implement the command to get container metrics

{code}
hdfs scm -container metrics 
{code}

this command returns container metrics in certain format, e.g json.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11677) OZone: SCM CLI: Implement get container command

[Weiwei Yang (JIRA)]

Weiwei Yang created HDFS-11677:
--

 Summary: OZone: SCM CLI: Implement get container command
 Key: HDFS-11677
 URL: https://issues.apache.org/jira/browse/HDFS-11677
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Reporter: Weiwei Yang


Implement get container

{code}
hdfs scm -container get  -o 
{code}

This command works only against a closed container. If the container is closed, 
then SCM will return the address of the datanodes. The datanodes support an API 
called copyCon- tainer, which returns the container as a tar ball.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11676) Ozone: SCM CLI: Implement close container command

[Weiwei Yang (JIRA)]

Weiwei Yang created HDFS-11676:
--

 Summary: Ozone: SCM CLI: Implement close container command
 Key: HDFS-11676
 URL: https://issues.apache.org/jira/browse/HDFS-11676
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Reporter: Weiwei Yang


Implement delete container

{code}
hdfs scm -container close 
{code}

This command connects to SCM and closes a container. Once the container is 
closed in the SCM, the corresponding container is closed at the appropriate 
datanode. if the container does not exist, it will return an error.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11675) Ozone: SCM CLI: Implement delete container command

[Weiwei Yang (JIRA)]

Weiwei Yang created HDFS-11675:
--

 Summary: Ozone: SCM CLI: Implement delete container command
 Key: HDFS-11675
 URL: https://issues.apache.org/jira/browse/HDFS-11675
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Reporter: Weiwei Yang


Implement delete container

{code}
hdfs scm -container del  -f
{code}

Deletes a container if it is empty. The -f options can be used to force a 
delete of a non-empty container. If container name specified not exist, prints 
a clear error message.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11674) reserveSpaceForReplicas is not released if append request failed due to mirror down and replica recovered

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-11674:


 Summary: reserveSpaceForReplicas is not released if append request 
failed due to mirror down and replica recovered
 Key: HDFS-11674
 URL: https://issues.apache.org/jira/browse/HDFS-11674
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode
Reporter: Vinayakumar B
Assignee: Vinayakumar B


Scenario:
1. 3 Node cluster with 
"dfs.client.block.write.replace-datanode-on-failure.policy"  as DEFAULT
Block is written with x data.
2. One of the Datanode, NOT the first DN, is down
3. Client tries to append data to block and fails since one DN is down.
4. calls recoverLease() on the file.
5. Successfull recovery happens.

Issue:
1. DNs which were connected from client before encountering mirror down, will 
have the reservedSpaceForReplicas incremented, BUT never decremented. 
2. So in long run DN's all space will be in reservedSpaceForReplicas resulting 
OutOfSpace errors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org