[jira] [Commented] (HDFS-7496) Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances
[ https://issues.apache.org/jira/browse/HDFS-7496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318982#comment-14318982 ] Colin Patrick McCabe commented on HDFS-7496: Good find, [~cnauroth]! Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances --- Key: HDFS-7496 URL: https://issues.apache.org/jira/browse/HDFS-7496 Project: Hadoop HDFS Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Lei (Eddy) Xu Fix For: 2.7.0 Attachments: HDFS-7496-branch-2.000.patch, HDFS-7496.000.patch, HDFS-7496.001.patch, HDFS-7496.002.patch, HDFS-7496.003.patch, HDFS-7496.003.patch, HDFS-7496.004.patch, HDFS-7496.005.patch, HDFS-7496.006.patch, HDFS-7496.007.patch We discussed a few FsVolume removal race conditions on the DataNode in HDFS-7489. We should figure out a way to make removing an FsVolume safe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7496) Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances
[
https://issues.apache.org/jira/browse/HDFS-7496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316761#comment-14316761
]
Chris Nauroth commented on HDFS-7496:
-
The trunk commit for this patch (git hash
b7f4a3156c0f5c600816c469637237ba6c9b330c) included a test class named
{{FsVolumeListTest}}. It appears that this class was not included in the
cherry-pick to branch-2 (git hash 5ec2b6caa9d63123a88f407f734319d4ac6038a9).
Can one of the original contributors or reviewers please clarify whether or not
this class was intended to be committed? I ask because it's causing a merge
conflict now for HDFS-7604.
Also, if the intention is to keep this class, then it will need to be renamed.
It has not actually been running in pre-commit. That's because our
maven-surefire-plugin configuration in hadoop-project/pom.xml is set to match
only on test classes beginning with Test in the name:
{code}
includes
include**/Test*.java/include
/includes
{code}
This is done to prevent the plugin from erroneously trying to run helper
classes under src/test/java as if they were JUnit suites.
Fix FsVolume removal race conditions on the DataNode by reference-counting
the volume instances
---
Key: HDFS-7496
URL: https://issues.apache.org/jira/browse/HDFS-7496
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Lei (Eddy) Xu
Fix For: 2.7.0
Attachments: HDFS-7496-branch-2.000.patch, HDFS-7496.000.patch,
HDFS-7496.001.patch, HDFS-7496.002.patch, HDFS-7496.003.patch,
HDFS-7496.003.patch, HDFS-7496.004.patch, HDFS-7496.005.patch,
HDFS-7496.006.patch, HDFS-7496.007.patch
We discussed a few FsVolume removal race conditions on the DataNode in
HDFS-7489. We should figure out a way to make removing an FsVolume safe.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7496) Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances
[
https://issues.apache.org/jira/browse/HDFS-7496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316854#comment-14316854
]
Lei (Eddy) Xu commented on HDFS-7496:
-
Hi, [~cnauroth]
This was my fault to misname the {{FsVolumeListTest}} and not include this test
in branch-2. Very sorry for the inconvenience. Shall I make a patch to correct
it in this JIRA or file a follow JIRA?
Fix FsVolume removal race conditions on the DataNode by reference-counting
the volume instances
---
Key: HDFS-7496
URL: https://issues.apache.org/jira/browse/HDFS-7496
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Lei (Eddy) Xu
Fix For: 2.7.0
Attachments: HDFS-7496-branch-2.000.patch, HDFS-7496.000.patch,
HDFS-7496.001.patch, HDFS-7496.002.patch, HDFS-7496.003.patch,
HDFS-7496.003.patch, HDFS-7496.004.patch, HDFS-7496.005.patch,
HDFS-7496.006.patch, HDFS-7496.007.patch
We discussed a few FsVolume removal race conditions on the DataNode in
HDFS-7489. We should figure out a way to make removing an FsVolume safe.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7496) Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances
[ https://issues.apache.org/jira/browse/HDFS-7496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316868#comment-14316868 ] Chris Nauroth commented on HDFS-7496: - [~eddyxu], thanks for the response. Let's handle it in a new jira, since this one has been closed for a while. Please feel free to contact me on the new jira for code review. Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances --- Key: HDFS-7496 URL: https://issues.apache.org/jira/browse/HDFS-7496 Project: Hadoop HDFS Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Lei (Eddy) Xu Fix For: 2.7.0 Attachments: HDFS-7496-branch-2.000.patch, HDFS-7496.000.patch, HDFS-7496.001.patch, HDFS-7496.002.patch, HDFS-7496.003.patch, HDFS-7496.003.patch, HDFS-7496.004.patch, HDFS-7496.005.patch, HDFS-7496.006.patch, HDFS-7496.007.patch We discussed a few FsVolume removal race conditions on the DataNode in HDFS-7489. We should figure out a way to make removing an FsVolume safe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7496) Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances
[ https://issues.apache.org/jira/browse/HDFS-7496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285780#comment-14285780 ] Hudson commented on HDFS-7496: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2031 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2031/]) HDFS-7496. Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances (lei via cmccabe) (cmccabe: rev b7f4a3156c0f5c600816c469637237ba6c9b330c) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaHandler.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeReference.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeListTest.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/ReplicaInputStreams.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteBlockGetsBlockLengthHint.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java HDFS-7496: add to CHANGES.txt (cmccabe: rev 73b72a048f70c275051747d13dc948845f4cef17) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances --- Key: HDFS-7496 URL: https://issues.apache.org/jira/browse/HDFS-7496 Project: Hadoop HDFS Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Lei (Eddy) Xu Fix For: 3.0.0 Attachments: HDFS-7496.000.patch, HDFS-7496.001.patch, HDFS-7496.002.patch, HDFS-7496.003.patch, HDFS-7496.003.patch, HDFS-7496.004.patch, HDFS-7496.005.patch, HDFS-7496.006.patch, HDFS-7496.007.patch We discussed a few FsVolume removal race conditions on the DataNode in HDFS-7489. We should figure out a way to make removing an FsVolume safe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7496) Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances
[ https://issues.apache.org/jira/browse/HDFS-7496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285522#comment-14285522 ] Hudson commented on HDFS-7496: -- FAILURE: Integrated in Hadoop-Yarn-trunk #814 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/814/]) HDFS-7496. Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances (lei via cmccabe) (cmccabe: rev b7f4a3156c0f5c600816c469637237ba6c9b330c) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeReference.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaHandler.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteBlockGetsBlockLengthHint.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeListTest.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/ReplicaInputStreams.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java HDFS-7496: add to CHANGES.txt (cmccabe: rev 73b72a048f70c275051747d13dc948845f4cef17) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances --- Key: HDFS-7496 URL: https://issues.apache.org/jira/browse/HDFS-7496 Project: Hadoop HDFS Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Lei (Eddy) Xu Fix For: 3.0.0 Attachments: HDFS-7496.000.patch, HDFS-7496.001.patch, HDFS-7496.002.patch, HDFS-7496.003.patch, HDFS-7496.003.patch, HDFS-7496.004.patch, HDFS-7496.005.patch, HDFS-7496.006.patch, HDFS-7496.007.patch We discussed a few FsVolume removal race conditions on the DataNode in HDFS-7489. We should figure out a way to make removing an FsVolume safe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7496) Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances
[ https://issues.apache.org/jira/browse/HDFS-7496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285539#comment-14285539 ] Hudson commented on HDFS-7496: -- SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #80 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/80/]) HDFS-7496. Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances (lei via cmccabe) (cmccabe: rev b7f4a3156c0f5c600816c469637237ba6c9b330c) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/ReplicaInputStreams.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteBlockGetsBlockLengthHint.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeListTest.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaHandler.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeReference.java HDFS-7496: add to CHANGES.txt (cmccabe: rev 73b72a048f70c275051747d13dc948845f4cef17) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances --- Key: HDFS-7496 URL: https://issues.apache.org/jira/browse/HDFS-7496 Project: Hadoop HDFS Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Lei (Eddy) Xu Fix For: 3.0.0 Attachments: HDFS-7496.000.patch, HDFS-7496.001.patch, HDFS-7496.002.patch, HDFS-7496.003.patch, HDFS-7496.003.patch, HDFS-7496.004.patch, HDFS-7496.005.patch, HDFS-7496.006.patch, HDFS-7496.007.patch We discussed a few FsVolume removal race conditions on the DataNode in HDFS-7489. We should figure out a way to make removing an FsVolume safe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7496) Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances
[ https://issues.apache.org/jira/browse/HDFS-7496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285724#comment-14285724 ] Hudson commented on HDFS-7496: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #81 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/81/]) HDFS-7496. Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances (lei via cmccabe) (cmccabe: rev b7f4a3156c0f5c600816c469637237ba6c9b330c) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaHandler.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteBlockGetsBlockLengthHint.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeListTest.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeReference.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/ReplicaInputStreams.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java HDFS-7496: add to CHANGES.txt (cmccabe: rev 73b72a048f70c275051747d13dc948845f4cef17) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances --- Key: HDFS-7496 URL: https://issues.apache.org/jira/browse/HDFS-7496 Project: Hadoop HDFS Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Lei (Eddy) Xu Fix For: 3.0.0 Attachments: HDFS-7496.000.patch, HDFS-7496.001.patch, HDFS-7496.002.patch, HDFS-7496.003.patch, HDFS-7496.003.patch, HDFS-7496.004.patch, HDFS-7496.005.patch, HDFS-7496.006.patch, HDFS-7496.007.patch We discussed a few FsVolume removal race conditions on the DataNode in HDFS-7489. We should figure out a way to make removing an FsVolume safe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7496) Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances
[ https://issues.apache.org/jira/browse/HDFS-7496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285667#comment-14285667 ] Hudson commented on HDFS-7496: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #77 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/77/]) HDFS-7496. Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances (lei via cmccabe) (cmccabe: rev b7f4a3156c0f5c600816c469637237ba6c9b330c) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeReference.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteBlockGetsBlockLengthHint.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/ReplicaInputStreams.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeListTest.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaHandler.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java HDFS-7496: add to CHANGES.txt (cmccabe: rev 73b72a048f70c275051747d13dc948845f4cef17) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances --- Key: HDFS-7496 URL: https://issues.apache.org/jira/browse/HDFS-7496 Project: Hadoop HDFS Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Lei (Eddy) Xu Fix For: 3.0.0 Attachments: HDFS-7496.000.patch, HDFS-7496.001.patch, HDFS-7496.002.patch, HDFS-7496.003.patch, HDFS-7496.003.patch, HDFS-7496.004.patch, HDFS-7496.005.patch, HDFS-7496.006.patch, HDFS-7496.007.patch We discussed a few FsVolume removal race conditions on the DataNode in HDFS-7489. We should figure out a way to make removing an FsVolume safe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7496) Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances
[ https://issues.apache.org/jira/browse/HDFS-7496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285686#comment-14285686 ] Hudson commented on HDFS-7496: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2012 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2012/]) HDFS-7496. Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances (lei via cmccabe) (cmccabe: rev b7f4a3156c0f5c600816c469637237ba6c9b330c) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeListTest.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeReference.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/ReplicaInputStreams.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteBlockGetsBlockLengthHint.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaHandler.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java HDFS-7496: add to CHANGES.txt (cmccabe: rev 73b72a048f70c275051747d13dc948845f4cef17) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances --- Key: HDFS-7496 URL: https://issues.apache.org/jira/browse/HDFS-7496 Project: Hadoop HDFS Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Lei (Eddy) Xu Fix For: 3.0.0 Attachments: HDFS-7496.000.patch, HDFS-7496.001.patch, HDFS-7496.002.patch, HDFS-7496.003.patch, HDFS-7496.003.patch, HDFS-7496.004.patch, HDFS-7496.005.patch, HDFS-7496.006.patch, HDFS-7496.007.patch We discussed a few FsVolume removal race conditions on the DataNode in HDFS-7489. We should figure out a way to make removing an FsVolume safe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7496) Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances
[ https://issues.apache.org/jira/browse/HDFS-7496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285119#comment-14285119 ] Hudson commented on HDFS-7496: -- SUCCESS: Integrated in Hadoop-trunk-Commit #6898 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6898/]) HDFS-7496. Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances (lei via cmccabe) (cmccabe: rev b7f4a3156c0f5c600816c469637237ba6c9b330c) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeReference.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/ReplicaInputStreams.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeListTest.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteBlockGetsBlockLengthHint.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaHandler.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java HDFS-7496: add to CHANGES.txt (cmccabe: rev 73b72a048f70c275051747d13dc948845f4cef17) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances --- Key: HDFS-7496 URL: https://issues.apache.org/jira/browse/HDFS-7496 Project: Hadoop HDFS Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Lei (Eddy) Xu Attachments: HDFS-7496.000.patch, HDFS-7496.001.patch, HDFS-7496.002.patch, HDFS-7496.003.patch, HDFS-7496.003.patch, HDFS-7496.004.patch, HDFS-7496.005.patch, HDFS-7496.006.patch, HDFS-7496.007.patch We discussed a few FsVolume removal race conditions on the DataNode in HDFS-7489. We should figure out a way to make removing an FsVolume safe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
