[jira] [Assigned] (HDDS-2301) Write path: Reduce read contention in rocksDB
[
https://issues.apache.org/jira/browse/HDDS-2301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh reassigned HDDS-2301:
---
Assignee: Supratim Deka (was: Nanda kumar)
> Write path: Reduce read contention in rocksDB
> -
>
> Key: HDDS-2301
> URL: https://issues.apache.org/jira/browse/HDDS-2301
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 0.5.0
>Reporter: Rajesh Balamohan
>Assignee: Supratim Deka
>Priority: Major
> Labels: performance
> Attachments: om_write_profile.png
>
>
> Benchmark:
>
> Simple benchmark which creates 100 and 1000s of keys (empty directory) in
> OM. This is done in a tight loop and multiple threads from client side to add
> enough load on CPU. Note that intention is to understand the bottlenecks in
> OM (intentionally avoiding interactions with SCM & DN).
> Observation:
> -
> During write path, Ozone checks {{OMFileRequest.verifyFilesInPath}}. This
> internally calls {{omMetadataManager.getKeyTable().get(dbKeyName)}} for every
> write operation. This turns out to be expensive and chokes the write path.
> [https://github.com/apache/hadoop/blob/trunk/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMDirectoryCreateRequest.java#L155]
> [https://github.com/apache/hadoop/blob/trunk/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java#L63]
> In most of the cases, directory creation would be fresh entry. In such cases,
> it would be good to try with {{RocksDB::keyMayExist.}}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14768) EC : Busy DN replica should be consider in live replica check.
[
https://issues.apache.org/jira/browse/HDFS-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956094#comment-16956094
]
guojh edited comment on HDFS-14768 at 10/21/19 1:44 PM:
[~surendrasingh] You can run the UT more times, the UT passed is because in the
process of decommission, the block in busyNode is correct, another block which
reconstruct is incorrect, NN find the block is redundant then will delete one,
NN may delete the correct one, may delete another one, If NN delete the
incorrect one, the UT is passed, if not, the UT can not passed. I am sorry, I
have no idea to get the incorrect block exactly。
was (Author: gjhkael):
[~surendrasingh] You can run the UT more times, the UT passed is because in the
process of decommission, the block in busyNode is correct another block
reconstruct is incorrect, redundant block will delete one, NN may delete the
correct one, may delete another one, If NN delete the incorrect one, the UT is
passed, if not, the UT can not passed. I am sorry, I have no idea to get the
incorrect block exactly。
> EC : Busy DN replica should be consider in live replica check.
> --
>
> Key: HDFS-14768
> URL: https://issues.apache.org/jira/browse/HDFS-14768
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding, hdfs, namenode
>Affects Versions: 3.0.2
>Reporter: guojh
>Assignee: guojh
>Priority: Major
> Labels: patch
> Fix For: 3.3.0
>
> Attachments: 1568275810244.jpg, 1568276338275.jpg, 1568771471942.jpg,
> HDFS-14768.000.patch, HDFS-14768.001.patch, HDFS-14768.002.patch,
> HDFS-14768.003.patch, HDFS-14768.004.patch, HDFS-14768.005.patch,
> HDFS-14768.006.patch, HDFS-14768.007.patch, HDFS-14768.jpg,
> guojh_UT_after_deomission.txt, guojh_UT_before_deomission.txt,
> zhaoyiming_UT_after_deomission.txt, zhaoyiming_UT_beofre_deomission.txt
>
>
> Policy is RS-6-3-1024K, version is hadoop 3.0.2;
> We suppose a file's block Index is [0,1,2,3,4,5,6,7,8], And decommission
> index[3,4], increase the index 6 datanode's
> pendingReplicationWithoutTargets that make it large than
> replicationStreamsHardLimit(we set 14). Then, After the method
> chooseSourceDatanodes of BlockMananger, the liveBlockIndices is
> [0,1,2,3,4,5,7,8], Block Counter is, Live:7, Decommission:2.
> In method scheduleReconstruction of BlockManager, the additionalReplRequired
> is 9 - 7 = 2. After Namenode choose two target Datanode, will assign a
> erasureCode task to target datanode.
> When datanode get the task will build targetIndices from liveBlockIndices
> and target length. the code is blow.
> {code:java}
> // code placeholder
> targetIndices = new short[targets.length];
> private void initTargetIndices() {
> BitSet bitset = reconstructor.getLiveBitSet();
> int m = 0; hasValidTargets = false;
> for (int i = 0; i < dataBlkNum + parityBlkNum; i++) {
> if (!bitset.get) {
> if (reconstructor.getBlockLen > 0) {
> if (m < targets.length) {
> targetIndices[m++] = (short)i;
> hasValidTargets = true;
> }
> }
> }
> }
> {code}
> targetIndices[0]=6, and targetIndices[1] is aways 0 from initial value.
> The StripedReader is aways create reader from first 6 index block, and is
> [0,1,2,3,4,5]
> Use the index [0,1,2,3,4,5] to build target index[6,0] will trigger the isal
> bug. the block index6's data is corruption(all data is zero).
> I write a unit test can stabilize repreduce.
> {code:java}
> // code placeholder
> private int replicationStreamsHardLimit =
> DFSConfigKeys.DFS_NAMENODE_REPLICATION_STREAMS_HARD_LIMIT_DEFAULT;
> numDNs = dataBlocks + parityBlocks + 10;
> @Test(timeout = 24)
> public void testFileDecommission() throws Exception {
> LOG.info("Starting test testFileDecommission");
> final Path ecFile = new Path(ecDir, "testFileDecommission");
> int writeBytes = cellSize * dataBlocks;
> writeStripedFile(dfs, ecFile, writeBytes);
> Assert.assertEquals(0, bm.numOfUnderReplicatedBlocks());
> FileChecksum fileChecksum1 = dfs.getFileChecksum(ecFile, writeBytes);
> final INodeFile fileNode = cluster.getNamesystem().getFSDirectory()
> .getINode4Write(ecFile.toString()).asFile();
> LocatedBlocks locatedBlocks =
> StripedFileTestUtil.getLocatedBlocks(ecFile, dfs);
> LocatedBlock lb = dfs.getClient().getLocatedBlocks(ecFile.toString(), 0)
> .get(0);
> DatanodeInfo[] dnLocs = lb.getLocations();
> LocatedStripedBlock lastBlock =
> (LocatedStripedBlock)locatedBlocks.getLastLocatedBlock();
> DatanodeInfo[] storageInfos = lastBlock.getLocations();
> //
> DatanodeDescriptor datanodeDescriptor =
> cluster.getNameNode().getNamesystem()
>
>
[jira] [Commented] (HDFS-14918) Remove useless getRedundancyThread from BlockManagerTestUtil
[
https://issues.apache.org/jira/browse/HDFS-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956100#comment-16956100
]
Hadoop QA commented on HDFS-14918:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 16s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14918 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12983607/HDFS-14918.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f30a75518eec 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 447f46d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/28144/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28144/testReport/ |
| Max. process+thread count | 3922 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/28144/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org
[jira] [Commented] (HDFS-14913) Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType()
[ https://issues.apache.org/jira/browse/HDFS-14913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956098#comment-16956098 ] Hudson commented on HDFS-14913: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17556 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17556/]) HDFS-14913. Correct the value of available count in (ayushsaxena: rev 74c2329fc36e0878555342085defb4e474ef1aad) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/net/TestDFSNetworkTopology.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/DFSNetworkTopology.java > Correct the value of available count in > DFSNetworkTopology#chooseRandomWithStorageType() > - > > Key: HDFS-14913 > URL: https://issues.apache.org/jira/browse/HDFS-14913 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Fix For: 3.3.0, 3.1.4, 3.2.2 > > Attachments: HDFS-14913-01.patch, HDFS-14913-02.patch, > HDFS-14913-03.patch > > > Presently if excluded scope is /default/rack1 and excluded node is > /default/rack10/node. Then the available count is not deducted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14768) EC : Busy DN replica should be consider in live replica check.
[
https://issues.apache.org/jira/browse/HDFS-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956094#comment-16956094
]
guojh commented on HDFS-14768:
--
[~surendrasingh] You can run the UT more times, the UT passed is because in the
process of decommission, the block in busyNode is correct another block
reconstruct is incorrect, redundant block will delete one, NN may delete the
correct one, may delete another one, If NN delete the incorrect one, the UT is
passed, if not, the UT can not passed. I am sorry, I have no idea to get the
incorrect block exactly。
> EC : Busy DN replica should be consider in live replica check.
> --
>
> Key: HDFS-14768
> URL: https://issues.apache.org/jira/browse/HDFS-14768
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding, hdfs, namenode
>Affects Versions: 3.0.2
>Reporter: guojh
>Assignee: guojh
>Priority: Major
> Labels: patch
> Fix For: 3.3.0
>
> Attachments: 1568275810244.jpg, 1568276338275.jpg, 1568771471942.jpg,
> HDFS-14768.000.patch, HDFS-14768.001.patch, HDFS-14768.002.patch,
> HDFS-14768.003.patch, HDFS-14768.004.patch, HDFS-14768.005.patch,
> HDFS-14768.006.patch, HDFS-14768.007.patch, HDFS-14768.jpg,
> guojh_UT_after_deomission.txt, guojh_UT_before_deomission.txt,
> zhaoyiming_UT_after_deomission.txt, zhaoyiming_UT_beofre_deomission.txt
>
>
> Policy is RS-6-3-1024K, version is hadoop 3.0.2;
> We suppose a file's block Index is [0,1,2,3,4,5,6,7,8], And decommission
> index[3,4], increase the index 6 datanode's
> pendingReplicationWithoutTargets that make it large than
> replicationStreamsHardLimit(we set 14). Then, After the method
> chooseSourceDatanodes of BlockMananger, the liveBlockIndices is
> [0,1,2,3,4,5,7,8], Block Counter is, Live:7, Decommission:2.
> In method scheduleReconstruction of BlockManager, the additionalReplRequired
> is 9 - 7 = 2. After Namenode choose two target Datanode, will assign a
> erasureCode task to target datanode.
> When datanode get the task will build targetIndices from liveBlockIndices
> and target length. the code is blow.
> {code:java}
> // code placeholder
> targetIndices = new short[targets.length];
> private void initTargetIndices() {
> BitSet bitset = reconstructor.getLiveBitSet();
> int m = 0; hasValidTargets = false;
> for (int i = 0; i < dataBlkNum + parityBlkNum; i++) {
> if (!bitset.get) {
> if (reconstructor.getBlockLen > 0) {
> if (m < targets.length) {
> targetIndices[m++] = (short)i;
> hasValidTargets = true;
> }
> }
> }
> }
> {code}
> targetIndices[0]=6, and targetIndices[1] is aways 0 from initial value.
> The StripedReader is aways create reader from first 6 index block, and is
> [0,1,2,3,4,5]
> Use the index [0,1,2,3,4,5] to build target index[6,0] will trigger the isal
> bug. the block index6's data is corruption(all data is zero).
> I write a unit test can stabilize repreduce.
> {code:java}
> // code placeholder
> private int replicationStreamsHardLimit =
> DFSConfigKeys.DFS_NAMENODE_REPLICATION_STREAMS_HARD_LIMIT_DEFAULT;
> numDNs = dataBlocks + parityBlocks + 10;
> @Test(timeout = 24)
> public void testFileDecommission() throws Exception {
> LOG.info("Starting test testFileDecommission");
> final Path ecFile = new Path(ecDir, "testFileDecommission");
> int writeBytes = cellSize * dataBlocks;
> writeStripedFile(dfs, ecFile, writeBytes);
> Assert.assertEquals(0, bm.numOfUnderReplicatedBlocks());
> FileChecksum fileChecksum1 = dfs.getFileChecksum(ecFile, writeBytes);
> final INodeFile fileNode = cluster.getNamesystem().getFSDirectory()
> .getINode4Write(ecFile.toString()).asFile();
> LocatedBlocks locatedBlocks =
> StripedFileTestUtil.getLocatedBlocks(ecFile, dfs);
> LocatedBlock lb = dfs.getClient().getLocatedBlocks(ecFile.toString(), 0)
> .get(0);
> DatanodeInfo[] dnLocs = lb.getLocations();
> LocatedStripedBlock lastBlock =
> (LocatedStripedBlock)locatedBlocks.getLastLocatedBlock();
> DatanodeInfo[] storageInfos = lastBlock.getLocations();
> //
> DatanodeDescriptor datanodeDescriptor =
> cluster.getNameNode().getNamesystem()
>
> .getBlockManager().getDatanodeManager().getDatanode(storageInfos[6].getDatanodeUuid());
> BlockInfo firstBlock = fileNode.getBlocks()[0];
> DatanodeStorageInfo[] dStorageInfos = bm.getStorages(firstBlock);
> // the first heartbeat will consume 3 replica tasks
> for (int i = 0; i <= replicationStreamsHardLimit + 3; i++) {
> BlockManagerTestUtil.addBlockToBeReplicated(datanodeDescriptor, new
> Block(i),
> new DatanodeStorageInfo[]{dStorageInfos[0]});
> }
> assertEquals(dataBlocks + parityBlocks, dnLocs.length);
> int[]
[jira] [Assigned] (HDFS-14884) Add sanity check that zone key equals feinfo key while setting Xattrs
[ https://issues.apache.org/jira/browse/HDFS-14884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh reassigned HDFS-14884: Assignee: Mukul Kumar Singh > Add sanity check that zone key equals feinfo key while setting Xattrs > - > > Key: HDFS-14884 > URL: https://issues.apache.org/jira/browse/HDFS-14884 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption, hdfs >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Attachments: HDFS-14884.001.patch, HDFS-14884.002.patch, > hdfs_distcp.patch > > > Currently, it is possible to set an external attribute where the zone key is > not the same as feinfo key. This jira will add a precondition before setting > this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956085#comment-16956085 ] Xiaoqiao He edited comment on HDFS-14882 at 10/21/19 1:31 PM: -- Check failed unit tests, most of them failed due to OOM, Please help to double check. Thanks [~ayushtkn], you are right. [^HDFS-14882.006.patch] remove unrelated changes. Please take another reviews. Thanks. was (Author: hexiaoqiao): Thanks [~ayushtkn], you are right. [^HDFS-14882.006.patch] remove unrelated changes. Please take another reviews. Thanks. > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, > HDFS-14882.006.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956085#comment-16956085 ] Xiaoqiao He commented on HDFS-14882: Thanks [~ayushtkn], you are right. [^HDFS-14882.006.patch] remove unrelated changes. Please take another reviews. Thanks. > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, > HDFS-14882.006.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[
https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956083#comment-16956083
]
Hadoop QA commented on HDFS-14882:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
52s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 46s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 521 unchanged - 1 fixed = 521 total (was 522) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 34s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}140m 11s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}207m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestSetrepDecreasing |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestSeekBug |
| | hadoop.hdfs.TestMaintenanceState |
| | hadoop.hdfs.TestReadStripedFileWithDNFailure |
| | hadoop.hdfs.TestDFSStartupVersions |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14882 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12983600/HDFS-14882.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 4183e6d38d44
[jira] [Updated] (HDFS-14913) Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType()
[ https://issues.apache.org/jira/browse/HDFS-14913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-14913: Fix Version/s: 3.2.2 3.1.4 3.3.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > Correct the value of available count in > DFSNetworkTopology#chooseRandomWithStorageType() > - > > Key: HDFS-14913 > URL: https://issues.apache.org/jira/browse/HDFS-14913 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Fix For: 3.3.0, 3.1.4, 3.2.2 > > Attachments: HDFS-14913-01.patch, HDFS-14913-02.patch, > HDFS-14913-03.patch > > > Presently if excluded scope is /default/rack1 and excluded node is > /default/rack10/node. Then the available count is not deducted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14913) Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType()
[ https://issues.apache.org/jira/browse/HDFS-14913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956078#comment-16956078 ] Ayush Saxena commented on HDFS-14913: - Committed to trunk, 3.2 and 3.1. Thanx [~leosun08] [~elgoiri] and [~surendrasingh] for the reviews!!! > Correct the value of available count in > DFSNetworkTopology#chooseRandomWithStorageType() > - > > Key: HDFS-14913 > URL: https://issues.apache.org/jira/browse/HDFS-14913 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14913-01.patch, HDFS-14913-02.patch, > HDFS-14913-03.patch > > > Presently if excluded scope is /default/rack1 and excluded node is > /default/rack10/node. Then the available count is not deducted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14882: --- Attachment: HDFS-14882.006.patch > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, > HDFS-14882.006.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2339) Add OzoneManager to MiniOzoneChaosCluster
Mukul Kumar Singh created HDDS-2339: --- Summary: Add OzoneManager to MiniOzoneChaosCluster Key: HDDS-2339 URL: https://issues.apache.org/jira/browse/HDDS-2339 Project: Hadoop Distributed Data Store Issue Type: Bug Components: om Reporter: Mukul Kumar Singh This jira proposes to add OzoneManager to MiniOzoneChaosCluster with OzoneHA implementation done. This will help in discovering bugs in Ozone Manager HA -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14916) RBF: line breaks are missing from the output of 'hdfs dfsrouteradmin -ls'
[
https://issues.apache.org/jira/browse/HDFS-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956065#comment-16956065
]
Hudson commented on HDFS-14916:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17555 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17555/])
HDFS-14916. RBF: line breaks are missing from the output of 'hdfs (ayushsaxena:
rev ff6a492dc9f8e15bde5d1bce3f8841298146)
* (edit)
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
> RBF: line breaks are missing from the output of 'hdfs dfsrouteradmin -ls'
> -
>
> Key: HDFS-14916
> URL: https://issues.apache.org/jira/browse/HDFS-14916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf, ui
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HDFS-14916.001.patch, HDFS-14916.patch
>
>
> line breaks seem to be missing from the output of "hdfs dfsrouteradmin -ls"
> e.g.:
> The output of "hdfs dfsrouteradmin -ls" now:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1 subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user2
> subCluster1->/tmp/user2user2 users
>rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user/user3
> subCluster1->/user/user3 user3 hadoop
> rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] [sri@nn00070 ~]$
> {code}
> This should be:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls -d
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user2subCluster1->/tmp/user2user2
> users rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user/user3 subCluster1->/user/user3 user3
> hadoop rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> [sri@nn00070 ~]$
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14916) RBF: line breaks are missing from the output of 'hdfs dfsrouteradmin -ls'
[
https://issues.apache.org/jira/browse/HDFS-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-14916:
Fix Version/s: 3.3.0
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk.
Thanx [~risyomei] for the contribution and [~tasanuma] for the review!!!
> RBF: line breaks are missing from the output of 'hdfs dfsrouteradmin -ls'
> -
>
> Key: HDFS-14916
> URL: https://issues.apache.org/jira/browse/HDFS-14916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf, ui
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HDFS-14916.001.patch, HDFS-14916.patch
>
>
> line breaks seem to be missing from the output of "hdfs dfsrouteradmin -ls"
> e.g.:
> The output of "hdfs dfsrouteradmin -ls" now:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1 subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user2
> subCluster1->/tmp/user2user2 users
>rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user/user3
> subCluster1->/user/user3 user3 hadoop
> rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] [sri@nn00070 ~]$
> {code}
> This should be:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls -d
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user2subCluster1->/tmp/user2user2
> users rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user/user3 subCluster1->/user/user3 user3
> hadoop rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> [sri@nn00070 ~]$
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14916) RBF: line breaks are missing from the output of 'hdfs dfsrouteradmin -ls'
[
https://issues.apache.org/jira/browse/HDFS-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-14916:
Summary: RBF: line breaks are missing from the output of 'hdfs
dfsrouteradmin -ls' (was: RBF: line breaks are missing from the output of
"hdfs dfsrouteradmin -ls")
> RBF: line breaks are missing from the output of 'hdfs dfsrouteradmin -ls'
> -
>
> Key: HDFS-14916
> URL: https://issues.apache.org/jira/browse/HDFS-14916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf, ui
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Minor
> Attachments: HDFS-14916.001.patch, HDFS-14916.patch
>
>
> line breaks seem to be missing from the output of "hdfs dfsrouteradmin -ls"
> e.g.:
> The output of "hdfs dfsrouteradmin -ls" now:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1 subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user2
> subCluster1->/tmp/user2user2 users
>rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user/user3
> subCluster1->/user/user3 user3 hadoop
> rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] [sri@nn00070 ~]$
> {code}
> This should be:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls -d
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user2subCluster1->/tmp/user2user2
> users rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user/user3 subCluster1->/user/user3 user3
> hadoop rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> [sri@nn00070 ~]$
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14916) RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
[
https://issues.apache.org/jira/browse/HDFS-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956035#comment-16956035
]
Ayush Saxena commented on HDFS-14916:
-
Thanx [~risyomei] for the patch.
v001 LGTM +1
> RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
> -
>
> Key: HDFS-14916
> URL: https://issues.apache.org/jira/browse/HDFS-14916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf, ui
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Minor
> Attachments: HDFS-14916.001.patch, HDFS-14916.patch
>
>
> line breaks seem to be missing from the output of "hdfs dfsrouteradmin -ls"
> e.g.:
> The output of "hdfs dfsrouteradmin -ls" now:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1 subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user2
> subCluster1->/tmp/user2user2 users
>rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user/user3
> subCluster1->/user/user3 user3 hadoop
> rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] [sri@nn00070 ~]$
> {code}
> This should be:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls -d
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user2subCluster1->/tmp/user2user2
> users rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user/user3 subCluster1->/user/user3 user3
> hadoop rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> [sri@nn00070 ~]$
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[
https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956033#comment-16956033
]
Ayush Saxena commented on HDFS-14882:
-
Thanx [~hexiaoqiao] for the patch.
{code:java}
-
-dfs.namenode.redundancy.considerLoad.factor
-2.0
-The factor by which a node's load can exceed the average
- before being rejected for writes, only if considerLoad is true.
-
-
+
+ dfs.namenode.redundancy.considerLoad.factor
+ 2.0
+ The factor by which a node's load can exceed the average
+before being rejected for writes, only if considerLoad is true.
+
+
{code}
This is not related to the change here, Just change in indentation, May be we
should avoid it.
Apart LGTM
> Consider DataNode load when #getBlockLocation
> -
>
> Key: HDFS-14882
> URL: https://issues.apache.org/jira/browse/HDFS-14882
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch,
> HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch
>
>
> Currently, we consider load of datanode when #chooseTarget for writer,
> however not consider it for reader. Thus, the process slot of datanode could
> be occupied by #BlockSender for reader, and disk/network will be busy
> workload, then meet some slow node exception. IIRC same case is reported
> times. Based on the fact, I propose to consider load for reader same as it
> did #chooseTarget for writer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13736) BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false
[
https://issues.apache.org/jira/browse/HDFS-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956027#comment-16956027
]
Xiaoqiao He commented on HDFS-13736:
[^HDFS-13736.004.patch] looks much better otherwise the following unnecessary
print line.
{quote}+ System.err.println(Arrays.toString(targets));
{quote}
Let's wait what Jenkins report {Test Results, checkstyle}, please check if the
failed unit test (failed unit test #TestRenameWithSnapshots last reported
looks not related) is related with changes and if any other checkstyle issues.
Please fix checkstyle first. Thanks.
Ping [~ayushtkn], would like to take another reviews?
> BlockPlacementPolicyDefault can not choose favored nodes when
> 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false
> --
>
> Key: HDFS-13736
> URL: https://issues.apache.org/jira/browse/HDFS-13736
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: hu xiaodong
>Assignee: hu xiaodong
>Priority: Major
> Attachments: HDFS-13736.001.patch, HDFS-13736.002.patch,
> HDFS-13736.003.patch, HDFS-13736.004.patch
>
>
> BlockPlacementPolicyDefault can not choose favored nodes when
> 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2331) Client OOME due to buffer retention
[
https://issues.apache.org/jira/browse/HDDS-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956020#comment-16956020
]
Attila Doroszlai commented on HDDS-2331:
Thanks [~shashikant] for the tip about setting chunk size to 1MB. By doing so,
writing 5K keys of size 1MB succeeds, even with 512MB max. heap. Taking a dump
at around 4K keys shows similar ~170 ContainerCommandRequestMessage instances.
(And yes, only half of the instances (writeChunk) have data buffer, the other
half is empty (putBlock)).
> Client OOME due to buffer retention
> ---
>
> Key: HDDS-2331
> URL: https://issues.apache.org/jira/browse/HDDS-2331
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.5.0
>Reporter: Attila Doroszlai
>Assignee: Shashikant Banerjee
>Priority: Critical
> Attachments: profiler.png
>
>
> Freon random key generator exhausts default heap after just few hundred 1MB
> keys. Heap dump on OOME reveals 150+ instances of
> {{ContainerCommandRequestMessage}}, each with 16MB {{byte[]}}.
> Steps to reproduce:
> # Start Ozone cluster with 1 datanode
> # Start Freon (5K keys of size 1MB)
> Result: OOME after a few hundred keys
> {noformat}
> $ cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone
> $ docker-compose up -d
> $ docker-compose exec scm bash
> $ export HADOOP_OPTS='-XX:+HeapDumpOnOutOfMemoryError'
> $ ozone freon rk --numOfThreads 1 --numOfVolumes 1 --numOfBuckets 1
> --replicationType RATIS --factor ONE --keySize 1048576 --numOfKeys 5120
> --bufferSize 65536
> ...
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid289.hprof ...
> Heap dump file created [1456141975 bytes in 7.760 secs]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14919) Provide Non DFS Used per disk in DataNode UI
[ https://issues.apache.org/jira/browse/HDFS-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14919: --- Attachment: hadoop3.1_datanode_ui.png hadoop2.6_datanode_ui.png > Provide Non DFS Used per disk in DataNode UI > > > Key: HDFS-14919 > URL: https://issues.apache.org/jira/browse/HDFS-14919 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Lisheng Sun >Priority: Major > Attachments: hadoop2.6_datanode_ui.png, hadoop3.1_datanode_ui.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14919) Provide Non DFS Used per disk in DataNode UI
Lisheng Sun created HDFS-14919: -- Summary: Provide Non DFS Used per disk in DataNode UI Key: HDFS-14919 URL: https://issues.apache.org/jira/browse/HDFS-14919 Project: Hadoop HDFS Issue Type: Bug Reporter: Lisheng Sun -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14802) The feature of protect directories should be used in RenameOp
[
https://issues.apache.org/jira/browse/HDFS-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956002#comment-16956002
]
Hadoop QA commented on HDFS-14802:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 22m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
8s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
5s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 41s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
48s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}241m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestAbandonBlock |
| | hadoop.hdfs.server.datanode.TestBlockRecovery |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14802 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12983584/HDFS-14802.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 6692ade31b43 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64
[jira] [Commented] (HDFS-13736) BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false
[ https://issues.apache.org/jira/browse/HDFS-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955994#comment-16955994 ] hu xiaodong commented on HDFS-13736: Hello, [~hexiaoqiao], Thanks for your advice. I submitted the fourth patch. Please help review. Thank you very much. > BlockPlacementPolicyDefault can not choose favored nodes when > 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false > -- > > Key: HDFS-13736 > URL: https://issues.apache.org/jira/browse/HDFS-13736 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.2.0 >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Major > Attachments: HDFS-13736.001.patch, HDFS-13736.002.patch, > HDFS-13736.003.patch, HDFS-13736.004.patch > > > BlockPlacementPolicyDefault can not choose favored nodes when > 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13736) BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false
[ https://issues.apache.org/jira/browse/HDFS-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hu xiaodong updated HDFS-13736: --- Attachment: HDFS-13736.004.patch > BlockPlacementPolicyDefault can not choose favored nodes when > 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false > -- > > Key: HDFS-13736 > URL: https://issues.apache.org/jira/browse/HDFS-13736 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.2.0 >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Major > Attachments: HDFS-13736.001.patch, HDFS-13736.002.patch, > HDFS-13736.003.patch, HDFS-13736.004.patch > > > BlockPlacementPolicyDefault can not choose favored nodes when > 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2331) Client OOME due to buffer retention
[
https://issues.apache.org/jira/browse/HDDS-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955983#comment-16955983
]
Tsz-wo Sze commented on HDDS-2331:
--
Chunk size and array size may not be necessarily the same. Chunk size (e.g.
16MB) is a larger value than array size (e.g. 1MB). A chunk could use a list
of arrays as the buffer.
> Client OOME due to buffer retention
> ---
>
> Key: HDDS-2331
> URL: https://issues.apache.org/jira/browse/HDDS-2331
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.5.0
>Reporter: Attila Doroszlai
>Assignee: Shashikant Banerjee
>Priority: Critical
> Attachments: profiler.png
>
>
> Freon random key generator exhausts default heap after just few hundred 1MB
> keys. Heap dump on OOME reveals 150+ instances of
> {{ContainerCommandRequestMessage}}, each with 16MB {{byte[]}}.
> Steps to reproduce:
> # Start Ozone cluster with 1 datanode
> # Start Freon (5K keys of size 1MB)
> Result: OOME after a few hundred keys
> {noformat}
> $ cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone
> $ docker-compose up -d
> $ docker-compose exec scm bash
> $ export HADOOP_OPTS='-XX:+HeapDumpOnOutOfMemoryError'
> $ ozone freon rk --numOfThreads 1 --numOfVolumes 1 --numOfBuckets 1
> --replicationType RATIS --factor ONE --keySize 1048576 --numOfKeys 5120
> --bufferSize 65536
> ...
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid289.hprof ...
> Heap dump file created [1456141975 bytes in 7.760 secs]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13736) BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false
[
https://issues.apache.org/jira/browse/HDFS-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955971#comment-16955971
]
Hadoop QA commented on HDFS-13736:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 6 new + 81 unchanged - 0 fixed = 87 total (was 81) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 31s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}149m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-13736 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12983590/HDFS-13736.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 0dacf3db8a96 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 447f46d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/28141/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/28141/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28141/testReport/ |
| Max. process+thread count | 4226 (vs. ulimit of 5500)
[jira] [Commented] (HDFS-14768) EC : Busy DN replica should be consider in live replica check.
[
https://issues.apache.org/jira/browse/HDFS-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955968#comment-16955968
]
Surendra Singh Lilhore commented on HDFS-14768:
---
[~gjhkael], Your UT is passing without fix also, pls can you check it once.
> EC : Busy DN replica should be consider in live replica check.
> --
>
> Key: HDFS-14768
> URL: https://issues.apache.org/jira/browse/HDFS-14768
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding, hdfs, namenode
>Affects Versions: 3.0.2
>Reporter: guojh
>Assignee: guojh
>Priority: Major
> Labels: patch
> Fix For: 3.3.0
>
> Attachments: 1568275810244.jpg, 1568276338275.jpg, 1568771471942.jpg,
> HDFS-14768.000.patch, HDFS-14768.001.patch, HDFS-14768.002.patch,
> HDFS-14768.003.patch, HDFS-14768.004.patch, HDFS-14768.005.patch,
> HDFS-14768.006.patch, HDFS-14768.007.patch, HDFS-14768.jpg,
> guojh_UT_after_deomission.txt, guojh_UT_before_deomission.txt,
> zhaoyiming_UT_after_deomission.txt, zhaoyiming_UT_beofre_deomission.txt
>
>
> Policy is RS-6-3-1024K, version is hadoop 3.0.2;
> We suppose a file's block Index is [0,1,2,3,4,5,6,7,8], And decommission
> index[3,4], increase the index 6 datanode's
> pendingReplicationWithoutTargets that make it large than
> replicationStreamsHardLimit(we set 14). Then, After the method
> chooseSourceDatanodes of BlockMananger, the liveBlockIndices is
> [0,1,2,3,4,5,7,8], Block Counter is, Live:7, Decommission:2.
> In method scheduleReconstruction of BlockManager, the additionalReplRequired
> is 9 - 7 = 2. After Namenode choose two target Datanode, will assign a
> erasureCode task to target datanode.
> When datanode get the task will build targetIndices from liveBlockIndices
> and target length. the code is blow.
> {code:java}
> // code placeholder
> targetIndices = new short[targets.length];
> private void initTargetIndices() {
> BitSet bitset = reconstructor.getLiveBitSet();
> int m = 0; hasValidTargets = false;
> for (int i = 0; i < dataBlkNum + parityBlkNum; i++) {
> if (!bitset.get) {
> if (reconstructor.getBlockLen > 0) {
> if (m < targets.length) {
> targetIndices[m++] = (short)i;
> hasValidTargets = true;
> }
> }
> }
> }
> {code}
> targetIndices[0]=6, and targetIndices[1] is aways 0 from initial value.
> The StripedReader is aways create reader from first 6 index block, and is
> [0,1,2,3,4,5]
> Use the index [0,1,2,3,4,5] to build target index[6,0] will trigger the isal
> bug. the block index6's data is corruption(all data is zero).
> I write a unit test can stabilize repreduce.
> {code:java}
> // code placeholder
> private int replicationStreamsHardLimit =
> DFSConfigKeys.DFS_NAMENODE_REPLICATION_STREAMS_HARD_LIMIT_DEFAULT;
> numDNs = dataBlocks + parityBlocks + 10;
> @Test(timeout = 24)
> public void testFileDecommission() throws Exception {
> LOG.info("Starting test testFileDecommission");
> final Path ecFile = new Path(ecDir, "testFileDecommission");
> int writeBytes = cellSize * dataBlocks;
> writeStripedFile(dfs, ecFile, writeBytes);
> Assert.assertEquals(0, bm.numOfUnderReplicatedBlocks());
> FileChecksum fileChecksum1 = dfs.getFileChecksum(ecFile, writeBytes);
> final INodeFile fileNode = cluster.getNamesystem().getFSDirectory()
> .getINode4Write(ecFile.toString()).asFile();
> LocatedBlocks locatedBlocks =
> StripedFileTestUtil.getLocatedBlocks(ecFile, dfs);
> LocatedBlock lb = dfs.getClient().getLocatedBlocks(ecFile.toString(), 0)
> .get(0);
> DatanodeInfo[] dnLocs = lb.getLocations();
> LocatedStripedBlock lastBlock =
> (LocatedStripedBlock)locatedBlocks.getLastLocatedBlock();
> DatanodeInfo[] storageInfos = lastBlock.getLocations();
> //
> DatanodeDescriptor datanodeDescriptor =
> cluster.getNameNode().getNamesystem()
>
> .getBlockManager().getDatanodeManager().getDatanode(storageInfos[6].getDatanodeUuid());
> BlockInfo firstBlock = fileNode.getBlocks()[0];
> DatanodeStorageInfo[] dStorageInfos = bm.getStorages(firstBlock);
> // the first heartbeat will consume 3 replica tasks
> for (int i = 0; i <= replicationStreamsHardLimit + 3; i++) {
> BlockManagerTestUtil.addBlockToBeReplicated(datanodeDescriptor, new
> Block(i),
> new DatanodeStorageInfo[]{dStorageInfos[0]});
> }
> assertEquals(dataBlocks + parityBlocks, dnLocs.length);
> int[] decommNodeIndex = {3, 4};
> final List decommisionNodes = new ArrayList();
> // add the node which will be decommissioning
> decommisionNodes.add(dnLocs[decommNodeIndex[0]]);
> decommisionNodes.add(dnLocs[decommNodeIndex[1]]);
> decommissionNode(0, decommisionNodes, AdminStates.DECOMMISSIONED);
[jira] [Assigned] (HDFS-14918) Remove useless getRedundancyThread from BlockManagerTestUtil
[
https://issues.apache.org/jira/browse/HDFS-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fei Hui reassigned HDFS-14918:
--
Assignee: Fei Hui
> Remove useless getRedundancyThread from BlockManagerTestUtil
>
>
> Key: HDFS-14918
> URL: https://issues.apache.org/jira/browse/HDFS-14918
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-14918.001.patch
>
>
> Remove the dead code, it's useless.
> {code}
> /**
> * @return redundancy monitor thread instance from block manager.
> */
> public static Daemon getRedundancyThread(final BlockManager blockManager) {
>return blockManager.getRedundancyThread();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14918) Remove useless getRedundancyThread from BlockManagerTestUtil
[
https://issues.apache.org/jira/browse/HDFS-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fei Hui updated HDFS-14918:
---
Attachment: HDFS-14918.001.patch
> Remove useless getRedundancyThread from BlockManagerTestUtil
>
>
> Key: HDFS-14918
> URL: https://issues.apache.org/jira/browse/HDFS-14918
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Fei Hui
>Priority: Major
> Attachments: HDFS-14918.001.patch
>
>
> Remove the dead code, it's useless.
> {code}
> /**
> * @return redundancy monitor thread instance from block manager.
> */
> public static Daemon getRedundancyThread(final BlockManager blockManager) {
>return blockManager.getRedundancyThread();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14918) Remove useless getRedundancyThread from BlockManagerTestUtil
[
https://issues.apache.org/jira/browse/HDFS-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fei Hui updated HDFS-14918:
---
Status: Patch Available (was: Open)
> Remove useless getRedundancyThread from BlockManagerTestUtil
>
>
> Key: HDFS-14918
> URL: https://issues.apache.org/jira/browse/HDFS-14918
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-14918.001.patch
>
>
> Remove the dead code, it's useless.
> {code}
> /**
> * @return redundancy monitor thread instance from block manager.
> */
> public static Daemon getRedundancyThread(final BlockManager blockManager) {
>return blockManager.getRedundancyThread();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14918) Remove useless getRedundancyThread from BlockManagerTestUtil
Fei Hui created HDFS-14918:
--
Summary: Remove useless getRedundancyThread from
BlockManagerTestUtil
Key: HDFS-14918
URL: https://issues.apache.org/jira/browse/HDFS-14918
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Fei Hui
Remove the dead code, it's useless.
{code}
/**
* @return redundancy monitor thread instance from block manager.
*/
public static Daemon getRedundancyThread(final BlockManager blockManager) {
return blockManager.getRedundancyThread();
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14915) Move Superuser Check Before Taking Lock For Encryption API
[
https://issues.apache.org/jira/browse/HDFS-14915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955947#comment-16955947
]
Hadoop QA commented on HDFS-14915:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 28m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 4s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 40s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
55s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}187m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14915 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12983582/HDFS-14915-02.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 1693e0370099 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 447f46d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/28139/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28139/testReport/ |
| Max.
[jira] [Commented] (HDFS-14917) Change the ICON of "Decommissioned & dead" datanode on "dfshealth.html"
[
https://issues.apache.org/jira/browse/HDFS-14917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955942#comment-16955942
]
Hadoop QA commented on HDFS-14917:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
32m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 48s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14917 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12983598/HDFS-14917.patch |
| Optional Tests | dupname asflicense shadedclient |
| uname | Linux 19389fda8e3b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 447f46d |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 450 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/28142/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Change the ICON of "Decommissioned & dead" datanode on "dfshealth.html"
> ---
>
> Key: HDFS-14917
> URL: https://issues.apache.org/jira/browse/HDFS-14917
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ui
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Trivial
> Attachments: HDFS-14917.patch, image-2019-10-21-17-49-10-635.png,
> image-2019-10-21-17-49-58-759.png, image-2019-10-21-18-03-53-914.png,
> image-2019-10-21-18-04-52-405.png, image-2019-10-21-18-05-19-160.png,
> image-2019-10-21-18-13-01-884.png, image-2019-10-21-18-13-54-427.png
>
>
> This is a really simple UI change proposal:
> The icon of "Decommissioned & dead" datanode could be improved. It can be
> changed from !image-2019-10-21-18-05-19-160.png|width=31,height=28! to
> !image-2019-10-21-18-04-52-405.png|width=32,height=29! so that,
> # icon " !image-2019-10-21-18-13-01-884.png|width=26,height=25! " can be
> used for all status starts with "decommission" on dfshealth.html,
> # icon " !image-2019-10-21-18-13-01-884.png|width=26,height=25! " can be
> differentiated with icon " !image-2019-10-21-18-13-54-427.png! " on
> federationhealth.html
> |*DataNode Infomation Legend (now)*
> dfshealth.html#tab-datanode
> |!image-2019-10-21-17-49-10-635.png|width=516,height=55!|
> |*DataNode* *Infomation* *Legend (proposed)*
> dfshealth.html#tab-datanode
> |!image-2019-10-21-18-03-53-914.png|width=589,height=60!|
> |*NameService Legend*
>
> federationhealth.htm#tab-namenode|!image-2019-10-21-17-49-58-759.png|width=445,height=43!|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11639) [PROVIDED Phase 2] Encode the BlockAlias in the client protocol
[
https://issues.apache.org/jira/browse/HDFS-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HDFS-11639:
--
Status: Open (was: Patch Available)
> [PROVIDED Phase 2] Encode the BlockAlias in the client protocol
> ---
>
> Key: HDFS-11639
> URL: https://issues.apache.org/jira/browse/HDFS-11639
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>Priority: Major
> Attachments: HDFS-11639-HDFS-9806.001.patch,
> HDFS-11639-HDFS-9806.002.patch, HDFS-11639-HDFS-9806.003.patch,
> HDFS-11639-HDFS-9806.004.patch, HDFS-11639-HDFS-9806.005.patch
>
>
> As part of the {{PROVIDED}} storage type, we have a {{BlockAlias}} type which
> encodes information about where the data comes from. i.e. URI, offset,
> length, and nonce value. This data should be encoded in the protocol
> ({{LocatedBlockProto}} and the {{BlockTokenIdentifier}}) when a block is
> available using the PROVIDED storage type.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11639) [PROVIDED Phase 2] Encode the BlockAlias in the client protocol
[
https://issues.apache.org/jira/browse/HDFS-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HDFS-11639:
--
Status: Patch Available (was: Open)
> [PROVIDED Phase 2] Encode the BlockAlias in the client protocol
> ---
>
> Key: HDFS-11639
> URL: https://issues.apache.org/jira/browse/HDFS-11639
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>Priority: Major
> Attachments: HDFS-11639-HDFS-9806.001.patch,
> HDFS-11639-HDFS-9806.002.patch, HDFS-11639-HDFS-9806.003.patch,
> HDFS-11639-HDFS-9806.004.patch, HDFS-11639-HDFS-9806.005.patch
>
>
> As part of the {{PROVIDED}} storage type, we have a {{BlockAlias}} type which
> encodes information about where the data comes from. i.e. URI, offset,
> length, and nonce value. This data should be encoded in the protocol
> ({{LocatedBlockProto}} and the {{BlockTokenIdentifier}}) when a block is
> available using the PROVIDED storage type.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11639) [PROVIDED Phase 2] Encode the BlockAlias in the client protocol
[
https://issues.apache.org/jira/browse/HDFS-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955929#comment-16955929
]
Ewan Higgs edited comment on HDFS-11639 at 10/21/19 10:03 AM:
--
https://github.com/apache/hadoop/pull/1665 submitted as a rebase of this work
onto HDFS-12090.
was (Author: ehiggs):
https://github.com/apache/hadoop/pull/1665 submitted as a rebase of this work.
> [PROVIDED Phase 2] Encode the BlockAlias in the client protocol
> ---
>
> Key: HDFS-11639
> URL: https://issues.apache.org/jira/browse/HDFS-11639
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>Priority: Major
> Attachments: HDFS-11639-HDFS-9806.001.patch,
> HDFS-11639-HDFS-9806.002.patch, HDFS-11639-HDFS-9806.003.patch,
> HDFS-11639-HDFS-9806.004.patch, HDFS-11639-HDFS-9806.005.patch
>
>
> As part of the {{PROVIDED}} storage type, we have a {{BlockAlias}} type which
> encodes information about where the data comes from. i.e. URI, offset,
> length, and nonce value. This data should be encoded in the protocol
> ({{LocatedBlockProto}} and the {{BlockTokenIdentifier}}) when a block is
> available using the PROVIDED storage type.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11639) [PROVIDED Phase 2] Encode the BlockAlias in the client protocol
[
https://issues.apache.org/jira/browse/HDFS-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955929#comment-16955929
]
Ewan Higgs commented on HDFS-11639:
---
https://github.com/apache/hadoop/pull/1665 submitted as a rebase of this work.
> [PROVIDED Phase 2] Encode the BlockAlias in the client protocol
> ---
>
> Key: HDFS-11639
> URL: https://issues.apache.org/jira/browse/HDFS-11639
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>Priority: Major
> Attachments: HDFS-11639-HDFS-9806.001.patch,
> HDFS-11639-HDFS-9806.002.patch, HDFS-11639-HDFS-9806.003.patch,
> HDFS-11639-HDFS-9806.004.patch, HDFS-11639-HDFS-9806.005.patch
>
>
> As part of the {{PROVIDED}} storage type, we have a {{BlockAlias}} type which
> encodes information about where the data comes from. i.e. URI, offset,
> length, and nonce value. This data should be encoded in the protocol
> ({{LocatedBlockProto}} and the {{BlockTokenIdentifier}}) when a block is
> available using the PROVIDED storage type.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14915) Move Superuser Check Before Taking Lock For Encryption API
[ https://issues.apache.org/jira/browse/HDFS-14915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955916#comment-16955916 ] Ayush Saxena edited comment on HDFS-14915 at 10/21/19 9:53 AM: --- Thanx [~hexiaoqiao] for the review, I think we can track the checkoperation in separate JIRA, Since I need to check the reasons for that before removing, and actually which one to remove. was (Author: ayushtkn): Thanx [~hexiaoqiao] for the review, I think we can track the checkoperation in separate JIRA > Move Superuser Check Before Taking Lock For Encryption API > -- > > Key: HDFS-14915 > URL: https://issues.apache.org/jira/browse/HDFS-14915 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14915-01.patch, HDFS-14915-02.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14915) Move Superuser Check Before Taking Lock For Encryption API
[ https://issues.apache.org/jira/browse/HDFS-14915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955919#comment-16955919 ] Xiaoqiao He commented on HDFS-14915: +1 for [^HDFS-14915-02.patch] from my side. > Move Superuser Check Before Taking Lock For Encryption API > -- > > Key: HDFS-14915 > URL: https://issues.apache.org/jira/browse/HDFS-14915 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14915-01.patch, HDFS-14915-02.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955918#comment-16955918 ] Xiaoqiao He commented on HDFS-14882: [^HDFS-14882.005.patch] fix checkstyle and try to trigger Jenkins again. > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14882: --- Attachment: HDFS-14882.005.patch > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14915) Move Superuser Check Before Taking Lock For Encryption API
[ https://issues.apache.org/jira/browse/HDFS-14915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955916#comment-16955916 ] Ayush Saxena commented on HDFS-14915: - Thanx [~hexiaoqiao] for the review, I think we can track the checkoperation in separate JIRA > Move Superuser Check Before Taking Lock For Encryption API > -- > > Key: HDFS-14915 > URL: https://issues.apache.org/jira/browse/HDFS-14915 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14915-01.patch, HDFS-14915-02.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[
https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955915#comment-16955915
]
Hadoop QA commented on HDFS-14882:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 56s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 521 unchanged - 1 fixed = 522 total (was 522) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}114m 57s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}187m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14882 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12983577/HDFS-14882.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 6ccbb7704940 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 447f46d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/28136/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
[jira] [Commented] (HDFS-13736) BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false
[
https://issues.apache.org/jira/browse/HDFS-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955911#comment-16955911
]
Xiaoqiao He commented on HDFS-13736:
Thanks [~xiaodong.hu] for interesting catching. [^HDFS-13736.003.patch] almost
LGTM. Some checkstyle nit comment.
1. line length should be better less than 80 characters. such as
#BlockPlacementPolicyDefault L244, L505, L555.
2. Please following alignment code style, such as the following
segment(#BlockPlacementPolicyDefault L590-594) may be better. Please also check
TestReplicationPolicy#testChooseFromFavoredNodesWhenPreferLocalSetToFalse or
let's wait what Jenkins says.
{code:java}
protected DatanodeStorageInfo chooseLocalStorage(Node localMachine,
boolean isFavoredNode, Set excludedNodes, long blocksize,
int maxNodesPerRack, List results,
boolean avoidStaleNodes, EnumMap storageTypes,
boolean fallbackToLocalRack) throws NotEnoughReplicasException {
{code}
3. It is better to add annotation for
BlockPlacementPolicyDefault#chooseLocalStorage, especially for new parameter
#isFavoredNode in order to avoid it's readability getting worse and worse.
Thanks [~xiaodong.hu] for your contributions again.
> BlockPlacementPolicyDefault can not choose favored nodes when
> 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false
> --
>
> Key: HDFS-13736
> URL: https://issues.apache.org/jira/browse/HDFS-13736
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: hu xiaodong
>Assignee: hu xiaodong
>Priority: Major
> Attachments: HDFS-13736.001.patch, HDFS-13736.002.patch,
> HDFS-13736.003.patch
>
>
> BlockPlacementPolicyDefault can not choose favored nodes when
> 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14917) Change the ICON of "Decommissioned & dead" datanode on "dfshealth.html"
[ https://issues.apache.org/jira/browse/HDFS-14917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xieming Li updated HDFS-14917: -- Attachment: HDFS-14917.patch Status: Patch Available (was: Open) > Change the ICON of "Decommissioned & dead" datanode on "dfshealth.html" > --- > > Key: HDFS-14917 > URL: https://issues.apache.org/jira/browse/HDFS-14917 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ui >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Trivial > Attachments: HDFS-14917.patch, image-2019-10-21-17-49-10-635.png, > image-2019-10-21-17-49-58-759.png, image-2019-10-21-18-03-53-914.png, > image-2019-10-21-18-04-52-405.png, image-2019-10-21-18-05-19-160.png, > image-2019-10-21-18-13-01-884.png, image-2019-10-21-18-13-54-427.png > > > This is a really simple UI change proposal: > The icon of "Decommissioned & dead" datanode could be improved. It can be > changed from !image-2019-10-21-18-05-19-160.png|width=31,height=28! to > !image-2019-10-21-18-04-52-405.png|width=32,height=29! so that, > # icon " !image-2019-10-21-18-13-01-884.png|width=26,height=25! " can be > used for all status starts with "decommission" on dfshealth.html, > # icon " !image-2019-10-21-18-13-01-884.png|width=26,height=25! " can be > differentiated with icon " !image-2019-10-21-18-13-54-427.png! " on > federationhealth.html > |*DataNode Infomation Legend (now)* > dfshealth.html#tab-datanode > |!image-2019-10-21-17-49-10-635.png|width=516,height=55!| > |*DataNode* *Infomation* *Legend (proposed)* > dfshealth.html#tab-datanode > |!image-2019-10-21-18-03-53-914.png|width=589,height=60!| > |*NameService Legend* > > federationhealth.htm#tab-namenode|!image-2019-10-21-17-49-58-759.png|width=445,height=43!| -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14917) Change the ICON of "Decommissioned & dead" datanode on "dfshealth.html"
[ https://issues.apache.org/jira/browse/HDFS-14917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xieming Li reassigned HDFS-14917: - Assignee: Xieming Li > Change the ICON of "Decommissioned & dead" datanode on "dfshealth.html" > --- > > Key: HDFS-14917 > URL: https://issues.apache.org/jira/browse/HDFS-14917 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ui >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Trivial > Attachments: image-2019-10-21-17-49-10-635.png, > image-2019-10-21-17-49-58-759.png, image-2019-10-21-18-03-53-914.png, > image-2019-10-21-18-04-52-405.png, image-2019-10-21-18-05-19-160.png, > image-2019-10-21-18-13-01-884.png, image-2019-10-21-18-13-54-427.png > > > This is a really simple UI change proposal: > The icon of "Decommissioned & dead" datanode could be improved. It can be > changed from !image-2019-10-21-18-05-19-160.png|width=31,height=28! to > !image-2019-10-21-18-04-52-405.png|width=32,height=29! so that, > # icon " !image-2019-10-21-18-13-01-884.png|width=26,height=25! " can be > used for all status starts with "decommission" on dfshealth.html, > # icon " !image-2019-10-21-18-13-01-884.png|width=26,height=25! " can be > differentiated with icon " !image-2019-10-21-18-13-54-427.png! " on > federationhealth.html > |*DataNode Infomation Legend (now)* > dfshealth.html#tab-datanode > |!image-2019-10-21-17-49-10-635.png|width=516,height=55!| > |*DataNode* *Infomation* *Legend (proposed)* > dfshealth.html#tab-datanode > |!image-2019-10-21-18-03-53-914.png|width=589,height=60!| > |*NameService Legend* > > federationhealth.htm#tab-namenode|!image-2019-10-21-17-49-58-759.png|width=445,height=43!| -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14917) Change the ICON of "Decommissioned & dead" datanode on "dfshealth.html"
[ https://issues.apache.org/jira/browse/HDFS-14917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xieming Li updated HDFS-14917: -- Description: This is a really simple UI change proposal: The icon of "Decommissioned & dead" datanode could be improved. It can be changed from !image-2019-10-21-18-05-19-160.png|width=31,height=28! to !image-2019-10-21-18-04-52-405.png|width=32,height=29! so that, # icon " !image-2019-10-21-18-13-01-884.png|width=26,height=25! " can be used for all status starts with "decommission" on dfshealth.html, # icon " !image-2019-10-21-18-13-01-884.png|width=26,height=25! " can be differentiated with icon " !image-2019-10-21-18-13-54-427.png! " on federationhealth.html |*DataNode Infomation Legend (now)* dfshealth.html#tab-datanode |!image-2019-10-21-17-49-10-635.png|width=516,height=55!| |*DataNode* *Infomation* *Legend (proposed)* dfshealth.html#tab-datanode |!image-2019-10-21-18-03-53-914.png|width=589,height=60!| |*NameService Legend* federationhealth.htm#tab-namenode|!image-2019-10-21-17-49-58-759.png|width=445,height=43!| was: This is a UI change proposal: The icon of "Decommissioned & dead" datanode could be improved. It can be changed from !image-2019-10-21-18-05-19-160.png|width=31,height=28! to !image-2019-10-21-18-04-52-405.png|width=32,height=29! so that, # icon " !image-2019-10-21-18-13-01-884.png|width=26,height=25! " can be used for all status starts with "decommission" on dfshealth.html, # icon " !image-2019-10-21-18-13-01-884.png|width=26,height=25! " can be differentiated with icon " !image-2019-10-21-18-13-54-427.png! " on federationhealth.html |*DataNode Infomation Legend* dfshealth.html#tab-datanode (now)|!image-2019-10-21-17-49-10-635.png|width=516,height=55!| |*DataNode* *Infomation* *Legend* dfshealth.html#tab-datanode (proposed)|!image-2019-10-21-18-03-53-914.png|width=589,height=60!| |*NameService Legend* federationhealth.htm#tab-namenode|!image-2019-10-21-17-49-58-759.png|width=445,height=43!| > Change the ICON of "Decommissioned & dead" datanode on "dfshealth.html" > --- > > Key: HDFS-14917 > URL: https://issues.apache.org/jira/browse/HDFS-14917 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ui >Reporter: Xieming Li >Priority: Trivial > Attachments: image-2019-10-21-17-49-10-635.png, > image-2019-10-21-17-49-58-759.png, image-2019-10-21-18-03-53-914.png, > image-2019-10-21-18-04-52-405.png, image-2019-10-21-18-05-19-160.png, > image-2019-10-21-18-13-01-884.png, image-2019-10-21-18-13-54-427.png > > > This is a really simple UI change proposal: > The icon of "Decommissioned & dead" datanode could be improved. It can be > changed from !image-2019-10-21-18-05-19-160.png|width=31,height=28! to > !image-2019-10-21-18-04-52-405.png|width=32,height=29! so that, > # icon " !image-2019-10-21-18-13-01-884.png|width=26,height=25! " can be > used for all status starts with "decommission" on dfshealth.html, > # icon " !image-2019-10-21-18-13-01-884.png|width=26,height=25! " can be > differentiated with icon " !image-2019-10-21-18-13-54-427.png! " on > federationhealth.html > |*DataNode Infomation Legend (now)* > dfshealth.html#tab-datanode > |!image-2019-10-21-17-49-10-635.png|width=516,height=55!| > |*DataNode* *Infomation* *Legend (proposed)* > dfshealth.html#tab-datanode > |!image-2019-10-21-18-03-53-914.png|width=589,height=60!| > |*NameService Legend* > > federationhealth.htm#tab-namenode|!image-2019-10-21-17-49-58-759.png|width=445,height=43!| -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14917) Change the ICON of "Decommissioned & dead" datanode on "dfshealth.html"
Xieming Li created HDFS-14917: - Summary: Change the ICON of "Decommissioned & dead" datanode on "dfshealth.html" Key: HDFS-14917 URL: https://issues.apache.org/jira/browse/HDFS-14917 Project: Hadoop HDFS Issue Type: Improvement Components: ui Reporter: Xieming Li Attachments: image-2019-10-21-17-49-10-635.png, image-2019-10-21-17-49-58-759.png, image-2019-10-21-18-03-53-914.png, image-2019-10-21-18-04-52-405.png, image-2019-10-21-18-05-19-160.png, image-2019-10-21-18-13-01-884.png, image-2019-10-21-18-13-54-427.png This is a UI change proposal: The icon of "Decommissioned & dead" datanode could be improved. It can be changed from !image-2019-10-21-18-05-19-160.png|width=31,height=28! to !image-2019-10-21-18-04-52-405.png|width=32,height=29! so that, # icon " !image-2019-10-21-18-13-01-884.png|width=26,height=25! " can be used for all status starts with "decommission" on dfshealth.html, # icon " !image-2019-10-21-18-13-01-884.png|width=26,height=25! " can be differentiated with icon " !image-2019-10-21-18-13-54-427.png! " on federationhealth.html |*DataNode Infomation Legend* dfshealth.html#tab-datanode (now)|!image-2019-10-21-17-49-10-635.png|width=516,height=55!| |*DataNode* *Infomation* *Legend* dfshealth.html#tab-datanode (proposed)|!image-2019-10-21-18-03-53-914.png|width=589,height=60!| |*NameService Legend* federationhealth.htm#tab-namenode|!image-2019-10-21-17-49-58-759.png|width=445,height=43!| -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14915) Move Superuser Check Before Taking Lock For Encryption API
[ https://issues.apache.org/jira/browse/HDFS-14915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955887#comment-16955887 ] Xiaoqiao He commented on HDFS-14915: Thanks [~ayushtkn] for your quick update. We need double #checkOperation both before and after hold lock. For #reencryptEncryptionZone, there are three times to do #checkOperation(WRITE) and distributed on three method. Some of them are duplicated. What I means is that we should delete some duplicated checking and it is better to embrace in one method (or file another JIRA to track). FYI. LGTM about superuser privilege checking improvement. > Move Superuser Check Before Taking Lock For Encryption API > -- > > Key: HDFS-14915 > URL: https://issues.apache.org/jira/browse/HDFS-14915 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14915-01.patch, HDFS-14915-02.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13736) BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false
[ https://issues.apache.org/jira/browse/HDFS-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955885#comment-16955885 ] hu xiaodong commented on HDFS-13736: Hello, [~ayushtkn]， I submitted the third patch as you said. can you help to have a look at it? Thank you very much. > BlockPlacementPolicyDefault can not choose favored nodes when > 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false > -- > > Key: HDFS-13736 > URL: https://issues.apache.org/jira/browse/HDFS-13736 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.2.0 >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Major > Attachments: HDFS-13736.001.patch, HDFS-13736.002.patch, > HDFS-13736.003.patch > > > BlockPlacementPolicyDefault can not choose favored nodes when > 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13736) BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false
[ https://issues.apache.org/jira/browse/HDFS-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hu xiaodong updated HDFS-13736: --- Attachment: HDFS-13736.003.patch > BlockPlacementPolicyDefault can not choose favored nodes when > 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false > -- > > Key: HDFS-13736 > URL: https://issues.apache.org/jira/browse/HDFS-13736 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.2.0 >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Major > Attachments: HDFS-13736.001.patch, HDFS-13736.002.patch, > HDFS-13736.003.patch > > > BlockPlacementPolicyDefault can not choose favored nodes when > 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14916) RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
[
https://issues.apache.org/jira/browse/HDFS-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955882#comment-16955882
]
Hadoop QA commented on HDFS-14916:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 46s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
9s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14916 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12983583/HDFS-14916.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 98d8d831a1c2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 447f46d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28138/testReport/ |
| Max. process+thread count | 2867 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/28138/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: line breaks are missing from the output
[jira] [Commented] (HDFS-14915) Move Superuser Check Before Taking Lock For Encryption API
[
https://issues.apache.org/jira/browse/HDFS-14915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955874#comment-16955874
]
Hadoop QA commented on HDFS-14915:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
48s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 56s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}110m 50s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}173m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14915 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12983571/HDFS-14915-01.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 3e89868747bb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 447f46d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/28135/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28135/testReport/ |
| Max. process+thread count | 2777 (vs. ulimit of
[jira] [Updated] (HDFS-6524) Choosing datanode retries times considering with block replica number
[ https://issues.apache.org/jira/browse/HDFS-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-6524: - Resolution: Not A Problem Status: Resolved (was: Patch Available) > Choosing datanode retries times considering with block replica number > -- > > Key: HDFS-6524 > URL: https://issues.apache.org/jira/browse/HDFS-6524 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0-alpha1 >Reporter: Liang Xie >Assignee: Lisheng Sun >Priority: Minor > Labels: BB2015-05-TBR > Attachments: HDFS-6524.001.patch, HDFS-6524.002.patch, > HDFS-6524.003.patch, HDFS-6524.004.patch, HDFS-6524.005(2).patch, > HDFS-6524.005.patch, HDFS-6524.006.patch, HDFS-6524.007.patch, HDFS-6524.txt > > > Currently the chooseDataNode() does retry with the setting: > dfsClientConf.maxBlockAcquireFailures, which by default is 3 > (DFS_CLIENT_MAX_BLOCK_ACQUIRE_FAILURES_DEFAULT = 3), it would be better > having another option, block replication factor. One cluster with only two > block replica setting, or using Reed-solomon encoding solution with one > replica factor. It helps to reduce the long tail latency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14913) Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType()
[ https://issues.apache.org/jira/browse/HDFS-14913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955865#comment-16955865 ] Surendra Singh Lilhore commented on HDFS-14913: --- +1 LGTM > Correct the value of available count in > DFSNetworkTopology#chooseRandomWithStorageType() > - > > Key: HDFS-14913 > URL: https://issues.apache.org/jira/browse/HDFS-14913 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14913-01.patch, HDFS-14913-02.patch, > HDFS-14913-03.patch > > > Presently if excluded scope is /default/rack1 and excluded node is > /default/rack10/node. Then the available count is not deducted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14753) Document HDFS-8432 (Introduce a minimum compatible layout version to allow downgrade in more rolling upgrade use cases)
[ https://issues.apache.org/jira/browse/HDFS-14753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955860#comment-16955860 ] Fei Hui commented on HDFS-14753: :( Document is difficult After HDFS-14509, RollingUpgrade from 2.x to 3.x has 3 steps: # upgrade from 2.x to 2.10 # upgrade from 2.10 to 3.x I think we can do some efforts for upgrading from 2.x directly to 3.x directly. During uprading NN generates password without new filelds, maybe we can do Upgrading from 2.x to 3.x directly HDFS-14831 Has not been fixed. it's need to be fixed because rollback is impossible because of incompatible changes. [~aajisaka] Any suggestions? > Document HDFS-8432 (Introduce a minimum compatible layout version to allow > downgrade in more rolling upgrade use cases) > --- > > Key: HDFS-14753 > URL: https://issues.apache.org/jira/browse/HDFS-14753 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation, rolling upgrades >Reporter: Akira Ajisaka >Priority: Major > > After HDFS-8432, we can downgrade to the different layout version if the > layout version of the destination is equal to or greater than the minimum > compatible layout version of the source. This information is very important > for administrators and it needs to be documented. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14753) Document HDFS-8432 (Introduce a minimum compatible layout version to allow downgrade in more rolling upgrade use cases)
[ https://issues.apache.org/jira/browse/HDFS-14753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955860#comment-16955860 ] Fei Hui edited comment on HDFS-14753 at 10/21/19 8:18 AM: -- :( Document is difficult After HDFS-14509, RollingUpgrade from 2.x to 3.x has 2 steps: # upgrade from 2.x to 2.10 # upgrade from 2.10 to 3.x I think we can do some efforts for upgrading from 2.x directly to 3.x directly. During uprading NN generates password without new filelds, maybe we can do Upgrading from 2.x to 3.x directly HDFS-14831 Has not been fixed. it's need to be fixed because rollback is impossible because of incompatible changes. [~aajisaka] Any suggestions? was (Author: ferhui): :( Document is difficult After HDFS-14509, RollingUpgrade from 2.x to 3.x has 3 steps: # upgrade from 2.x to 2.10 # upgrade from 2.10 to 3.x I think we can do some efforts for upgrading from 2.x directly to 3.x directly. During uprading NN generates password without new filelds, maybe we can do Upgrading from 2.x to 3.x directly HDFS-14831 Has not been fixed. it's need to be fixed because rollback is impossible because of incompatible changes. [~aajisaka] Any suggestions? > Document HDFS-8432 (Introduce a minimum compatible layout version to allow > downgrade in more rolling upgrade use cases) > --- > > Key: HDFS-14753 > URL: https://issues.apache.org/jira/browse/HDFS-14753 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation, rolling upgrades >Reporter: Akira Ajisaka >Priority: Major > > After HDFS-8432, we can downgrade to the different layout version if the > layout version of the destination is equal to or greater than the minimum > compatible layout version of the source. This information is very important > for administrators and it needs to be documented. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14916) RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
[
https://issues.apache.org/jira/browse/HDFS-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955857#comment-16955857
]
Hadoop QA commented on HDFS-14916:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m
54s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14916 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12983578/HDFS-14916.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d14cf122bf1f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 447f46d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28137/testReport/ |
| Max. process+thread count | 2900 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/28137/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: line breaks are missing from the output of
[jira] [Commented] (HDDS-2331) Client OOME due to buffer retention
[
https://issues.apache.org/jira/browse/HDDS-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955856#comment-16955856
]
Shashikant Banerjee commented on HDDS-2331:
---
One more point to add here is, as per the cmd, freon is run with a single
thread itself, which i assume will write only 1 key at a time. For a 1MB key,
we should have only at max 2 containerCommadRequestMsgs(one for write chunk and
one for putBlock) and these objects should disappear/garbage collected after
the response is received. At any point of time, it should not hence show up
150+ containerCommandRequestObjects lying around.
> Client OOME due to buffer retention
> ---
>
> Key: HDDS-2331
> URL: https://issues.apache.org/jira/browse/HDDS-2331
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.5.0
>Reporter: Attila Doroszlai
>Assignee: Shashikant Banerjee
>Priority: Critical
> Attachments: profiler.png
>
>
> Freon random key generator exhausts default heap after just few hundred 1MB
> keys. Heap dump on OOME reveals 150+ instances of
> {{ContainerCommandRequestMessage}}, each with 16MB {{byte[]}}.
> Steps to reproduce:
> # Start Ozone cluster with 1 datanode
> # Start Freon (5K keys of size 1MB)
> Result: OOME after a few hundred keys
> {noformat}
> $ cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone
> $ docker-compose up -d
> $ docker-compose exec scm bash
> $ export HADOOP_OPTS='-XX:+HeapDumpOnOutOfMemoryError'
> $ ozone freon rk --numOfThreads 1 --numOfVolumes 1 --numOfBuckets 1
> --replicationType RATIS --factor ONE --keySize 1048576 --numOfKeys 5120
> --bufferSize 65536
> ...
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid289.hprof ...
> Heap dump file created [1456141975 bytes in 7.760 secs]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14802) The feature of protect directories should be used in RenameOp
[ https://issues.apache.org/jira/browse/HDFS-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955846#comment-16955846 ] Fei Hui commented on HDFS-14802: [~ste...@apache.org] [~ayushtkn] Thanks. Upload v004 patch. * Mention protected dirs at filesystem.md * move method to DFSUtil * do the check before unprotected methods. > The feature of protect directories should be used in RenameOp > - > > Key: HDFS-14802 > URL: https://issues.apache.org/jira/browse/HDFS-14802 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.0.4, 3.3.0, 3.2.1, 3.1.3 >Reporter: Fei Hui >Assignee: Fei Hui >Priority: Major > Attachments: HDFS-14802.001.patch, HDFS-14802.002.patch, > HDFS-14802.003.patch, HDFS-14802.004.patch > > > Now we could set fs.protected.directories to prevent users from deleting > important directories. But users can delete directories around the limitation. > 1. Rename the directories and delete them. > 2. move the directories to trash and namenode will delete them. > So I think we should use the feature of protected directories in RenameOp -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14802) The feature of protect directories should be used in RenameOp
[ https://issues.apache.org/jira/browse/HDFS-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Hui updated HDFS-14802: --- Attachment: HDFS-14802.004.patch > The feature of protect directories should be used in RenameOp > - > > Key: HDFS-14802 > URL: https://issues.apache.org/jira/browse/HDFS-14802 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.0.4, 3.3.0, 3.2.1, 3.1.3 >Reporter: Fei Hui >Assignee: Fei Hui >Priority: Major > Attachments: HDFS-14802.001.patch, HDFS-14802.002.patch, > HDFS-14802.003.patch, HDFS-14802.004.patch > > > Now we could set fs.protected.directories to prevent users from deleting > important directories. But users can delete directories around the limitation. > 1. Rename the directories and delete them. > 2. move the directories to trash and namenode will delete them. > So I think we should use the feature of protected directories in RenameOp -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14916) RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
[
https://issues.apache.org/jira/browse/HDFS-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955834#comment-16955834
]
Xieming Li edited comment on HDFS-14916 at 10/21/19 7:48 AM:
-
[~ayushtkn], [~tasanuma],
Thank you for your confirmation.
I have updated the patch.
was (Author: risyomei):
[~ayushtkn], [~tasanuma],
thank you for your confirmation.
I have updated the patch.
> RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
> -
>
> Key: HDFS-14916
> URL: https://issues.apache.org/jira/browse/HDFS-14916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf, ui
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Minor
> Attachments: HDFS-14916.001.patch, HDFS-14916.patch
>
>
> line breaks seem to be missing from the output of "hdfs dfsrouteradmin -ls"
> e.g.:
> The output of "hdfs dfsrouteradmin -ls" now:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1 subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user2
> subCluster1->/tmp/user2user2 users
>rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user/user3
> subCluster1->/user/user3 user3 hadoop
> rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] [sri@nn00070 ~]$
> {code}
> This should be:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls -d
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user2subCluster1->/tmp/user2user2
> users rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user/user3 subCluster1->/user/user3 user3
> hadoop rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> [sri@nn00070 ~]$
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14916) RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
[
https://issues.apache.org/jira/browse/HDFS-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955837#comment-16955837
]
Takanobu Asanuma commented on HDFS-14916:
-
+1 on [^HDFS-14916.001.patch], pending Jenkins.
> RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
> -
>
> Key: HDFS-14916
> URL: https://issues.apache.org/jira/browse/HDFS-14916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf, ui
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Minor
> Attachments: HDFS-14916.001.patch, HDFS-14916.patch
>
>
> line breaks seem to be missing from the output of "hdfs dfsrouteradmin -ls"
> e.g.:
> The output of "hdfs dfsrouteradmin -ls" now:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1 subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user2
> subCluster1->/tmp/user2user2 users
>rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user/user3
> subCluster1->/user/user3 user3 hadoop
> rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] [sri@nn00070 ~]$
> {code}
> This should be:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls -d
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user2subCluster1->/tmp/user2user2
> users rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user/user3 subCluster1->/user/user3 user3
> hadoop rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> [sri@nn00070 ~]$
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14915) Move Superuser Check Before Taking Lock For Encryption API
[ https://issues.apache.org/jira/browse/HDFS-14915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955835#comment-16955835 ] Ayush Saxena commented on HDFS-14915: - Thanx [~hexiaoqiao] uploaded v2. removing the redundant superuser check. > Move Superuser Check Before Taking Lock For Encryption API > -- > > Key: HDFS-14915 > URL: https://issues.apache.org/jira/browse/HDFS-14915 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14915-01.patch, HDFS-14915-02.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14916) RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
[
https://issues.apache.org/jira/browse/HDFS-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955834#comment-16955834
]
Xieming Li commented on HDFS-14916:
---
[~ayushtkn], [~tasanuma],
thank you for your confirmation.
I have updated the patch.
> RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
> -
>
> Key: HDFS-14916
> URL: https://issues.apache.org/jira/browse/HDFS-14916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf, ui
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Minor
> Attachments: HDFS-14916.001.patch, HDFS-14916.patch
>
>
> line breaks seem to be missing from the output of "hdfs dfsrouteradmin -ls"
> e.g.:
> The output of "hdfs dfsrouteradmin -ls" now:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1 subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user2
> subCluster1->/tmp/user2user2 users
>rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user/user3
> subCluster1->/user/user3 user3 hadoop
> rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] [sri@nn00070 ~]$
> {code}
> This should be:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls -d
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user2subCluster1->/tmp/user2user2
> users rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user/user3 subCluster1->/user/user3 user3
> hadoop rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> [sri@nn00070 ~]$
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14916) RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
[
https://issues.apache.org/jira/browse/HDFS-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xieming Li updated HDFS-14916:
--
Attachment: HDFS-14916.001.patch
Status: Patch Available (was: Open)
> RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
> -
>
> Key: HDFS-14916
> URL: https://issues.apache.org/jira/browse/HDFS-14916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf, ui
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Minor
> Attachments: HDFS-14916.001.patch, HDFS-14916.patch
>
>
> line breaks seem to be missing from the output of "hdfs dfsrouteradmin -ls"
> e.g.:
> The output of "hdfs dfsrouteradmin -ls" now:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1 subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user2
> subCluster1->/tmp/user2user2 users
>rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user/user3
> subCluster1->/user/user3 user3 hadoop
> rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] [sri@nn00070 ~]$
> {code}
> This should be:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls -d
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user2subCluster1->/tmp/user2user2
> users rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user/user3 subCluster1->/user/user3 user3
> hadoop rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> [sri@nn00070 ~]$
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14915) Move Superuser Check Before Taking Lock For Encryption API
[ https://issues.apache.org/jira/browse/HDFS-14915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-14915: Attachment: HDFS-14915-02.patch > Move Superuser Check Before Taking Lock For Encryption API > -- > > Key: HDFS-14915 > URL: https://issues.apache.org/jira/browse/HDFS-14915 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14915-01.patch, HDFS-14915-02.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14916) RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
[
https://issues.apache.org/jira/browse/HDFS-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xieming Li updated HDFS-14916:
--
Status: Open (was: Patch Available)
> RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
> -
>
> Key: HDFS-14916
> URL: https://issues.apache.org/jira/browse/HDFS-14916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf, ui
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Minor
> Attachments: HDFS-14916.001.patch, HDFS-14916.patch
>
>
> line breaks seem to be missing from the output of "hdfs dfsrouteradmin -ls"
> e.g.:
> The output of "hdfs dfsrouteradmin -ls" now:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1 subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user2
> subCluster1->/tmp/user2user2 users
>rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user/user3
> subCluster1->/user/user3 user3 hadoop
> rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] [sri@nn00070 ~]$
> {code}
> This should be:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls -d
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user2subCluster1->/tmp/user2user2
> users rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user/user3 subCluster1->/user/user3 user3
> hadoop rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> [sri@nn00070 ~]$
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-669) Consider removing duplication of StorageType
[ https://issues.apache.org/jira/browse/HDDS-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YiSheng Lien reassigned HDDS-669: - Assignee: YiSheng Lien > Consider removing duplication of StorageType > > > Key: HDDS-669 > URL: https://issues.apache.org/jira/browse/HDDS-669 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Arpit Agarwal >Assignee: YiSheng Lien >Priority: Major > Labels: beta1 > > The StorageType class is currently duplicated in hadoop-hdds. We can just use > the version in hadoop-common. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-529) Some Ozone DataNode logs go to a separate ozone.log file
[ https://issues.apache.org/jira/browse/HDDS-529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YiSheng Lien reassigned HDDS-529: - Assignee: YiSheng Lien > Some Ozone DataNode logs go to a separate ozone.log file > > > Key: HDDS-529 > URL: https://issues.apache.org/jira/browse/HDDS-529 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Arpit Agarwal >Assignee: YiSheng Lien >Priority: Blocker > Labels: beta1 > > Some, but not all DataNode logs go to a separate ozone.log file. Couple of > things to fix here: > # The behavior should be consistent. All log messages should go to the new > log file. > # The new log file name should follow the Hadoop log file convention e.g. > _hadoop---.log_ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14916) RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
[
https://issues.apache.org/jira/browse/HDFS-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955824#comment-16955824
]
Takanobu Asanuma commented on HDFS-14916:
-
[~risyomei] Thanks for catching the issue.
IntelliJ warns the empty is unnecessary. We can use {{System.out.println()}}.
The Other looks good to me.
> RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
> -
>
> Key: HDFS-14916
> URL: https://issues.apache.org/jira/browse/HDFS-14916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf, ui
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Minor
> Attachments: HDFS-14916.patch
>
>
> line breaks seem to be missing from the output of "hdfs dfsrouteradmin -ls"
> e.g.:
> The output of "hdfs dfsrouteradmin -ls" now:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1 subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user2
> subCluster1->/tmp/user2user2 users
>rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user/user3
> subCluster1->/user/user3 user3 hadoop
> rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] [sri@nn00070 ~]$
> {code}
> This should be:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls -d
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user2subCluster1->/tmp/user2user2
> users rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user/user3 subCluster1->/user/user3 user3
> hadoop rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> [sri@nn00070 ~]$
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Deleted] (HDDS-2338) Avoid buffer copy while submitting write chunk request in Ozone Client
[ https://issues.apache.org/jira/browse/HDDS-2338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee deleted HDDS-2338: -- > Avoid buffer copy while submitting write chunk request in Ozone Client > -- > > Key: HDDS-2338 > URL: https://issues.apache.org/jira/browse/HDDS-2338 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > > Based, on the config value of "ozone.UnsafeByteOperations.enabled" which by > default is set to true we used to avoid buffer copy while submitting write > chunk request to Ratis. With recent changes around byteStringConversion > utility, seems like the config is never passed to BlockOutputStream and it > results in buffer copying every time there is a byteBuffer to byteString > conversion is done in ozone client. This Jira is to pass the appropriate > config value so that buffer copy can be avoided. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2338) Avoid buffer copy while submitting write chunk request in Ozone Client
[ https://issues.apache.org/jira/browse/HDDS-2338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee reassigned HDDS-2338: - Assignee: Shashikant Banerjee > Avoid buffer copy while submitting write chunk request in Ozone Client > -- > > Key: HDDS-2338 > URL: https://issues.apache.org/jira/browse/HDDS-2338 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.5.0 > > > Based, on the config value of "ozone.UnsafeByteOperations.enabled" which by > default is set to true we used to avoid buffer copy while submitting write > chunk request to Ratis. With recent changes around byteStringConversion > utility, seems like the config is never passed to BlockOutputStream and it > results in buffer copying every time there is a byteBuffer to byteString > conversion is done in ozone client. This Jira is to pass the appropriate > config value so that buffer copy can be avoided. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1576) Ensure constraint of one raft log per disk is met unless fast media
[ https://issues.apache.org/jira/browse/HDDS-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng reassigned HDDS-1576: -- Assignee: Li Cheng > Ensure constraint of one raft log per disk is met unless fast media > --- > > Key: HDDS-1576 > URL: https://issues.apache.org/jira/browse/HDDS-1576 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Datanode, SCM >Reporter: Siddharth Wagle >Assignee: Li Cheng >Priority: Major > > SCM should not try to create a raft group by placing the raft log on a disk > that is already used by existing Ratis ring for an open pipeline. > This constraint would have to be applied by either throwing an exception > during pipeline creation or by looking at configs on the SCM side. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2338) Avoid buffer copy while submitting write chunk request in Ozone Client
Shashikant Banerjee created HDDS-2338: - Summary: Avoid buffer copy while submitting write chunk request in Ozone Client Key: HDDS-2338 URL: https://issues.apache.org/jira/browse/HDDS-2338 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Reporter: Shashikant Banerjee Fix For: 0.5.0 Based, on the config value of "ozone.UnsafeByteOperations.enabled" which by default is set to true we used to avoid buffer copy while submitting write chunk request to Ratis. With recent changes around byteStringConversion utility, seems like the config is never passed to BlockOutputStream and it results in buffer copying every time there is a byteBuffer to byteString conversion is done in ozone client. This Jira is to pass the appropriate config value so that buffer copy can be avoided. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14916) RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
[
https://issues.apache.org/jira/browse/HDFS-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xieming Li updated HDFS-14916:
--
Attachment: HDFS-14916.patch
Status: Patch Available (was: Open)
> RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
> -
>
> Key: HDFS-14916
> URL: https://issues.apache.org/jira/browse/HDFS-14916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf, ui
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Minor
> Attachments: HDFS-14916.patch
>
>
> line breaks seem to be missing from the output of "hdfs dfsrouteradmin -ls"
> e.g.:
> The output of "hdfs dfsrouteradmin -ls" now:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1 subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user2
> subCluster1->/tmp/user2user2 users
>rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user/user3
> subCluster1->/user/user3 user3 hadoop
> rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] [sri@nn00070 ~]$
> {code}
> This should be:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls -d
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user2subCluster1->/tmp/user2user2
> users rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user/user3 subCluster1->/user/user3 user3
> hadoop rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> [sri@nn00070 ~]$
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14915) Move Superuser Check Before Taking Lock For Encryption API
[ https://issues.apache.org/jira/browse/HDFS-14915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955815#comment-16955815 ] Xiaoqiao He commented on HDFS-14915: [~ayushtkn] Thanks for your works. it seems that double check operations and superuser check distributed on different method (such as #reencryptEncryptionZone + #reencryptEncryptionZoneInt), some of them are duplicated. Would you like to refactor it for once? +1 for other changes. > Move Superuser Check Before Taking Lock For Encryption API > -- > > Key: HDFS-14915 > URL: https://issues.apache.org/jira/browse/HDFS-14915 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14915-01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-2331) Client OOME due to buffer retention
[
https://issues.apache.org/jira/browse/HDDS-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955814#comment-16955814
]
Shashikant Banerjee edited comment on HDDS-2331 at 10/21/19 7:04 AM:
-
In Ozone, by default the buffer size is equal to the chunk size(16 MB default).
Once a write call happens, a buffer is allocated and data is being written just
into the buffer till it gets full/flush/close and then pushed to datanode and
the buffer gets only released when watchForCommit call for the respective
putBlock call log Index succeeds successfully. So until and unless,
watchForCommit call gets acknowledged by Ozone Client, we keep holding onto the
buffer so that, in case the ratis request fails, we have the user data cached
in the client buffer which can be written over to the next block.
We have had multiple discussions around on reducing the default buffer size
and implement a true streaming client, but this is still under consideration.
[~adoroszlai], for your test, you can try changing the default chunk size to
say 1 MB and see if it works well. It might also be possible that buffer
release handling got broken with some changes introduced which need to be
verified.
was (Author: shashikant):
In Ozone, by default the buffer size is equal to the chunk size(16 MB default).
Once a write call happens, a buffer is allocated and data is being written just
into the buffer till it gets full/flush/close and then pushed to datanode and
the buffer gets only released when watchForCommit call for the respective
putBlock call log Index succeeds successfully. So until and unless,
watchForCommit call gets acknowledged by Ratis, we keep holding onto the buffer
so that, in case the ratis request fails, we have the user data cached in the
client buffer which can be written over to the next block.
We have had multiple discussions around this on reducing the default buffer
size and implement a true streaming client, but this is still under
consideration.
[~adoroszlai], for your test, you can try changing the default chunk size to
say 1 MB and see if it works well. It might also be possible that buffer
release handling got broken with some changes introduced which need to be
verified.
> Client OOME due to buffer retention
> ---
>
> Key: HDDS-2331
> URL: https://issues.apache.org/jira/browse/HDDS-2331
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.5.0
>Reporter: Attila Doroszlai
>Assignee: Shashikant Banerjee
>Priority: Critical
> Attachments: profiler.png
>
>
> Freon random key generator exhausts default heap after just few hundred 1MB
> keys. Heap dump on OOME reveals 150+ instances of
> {{ContainerCommandRequestMessage}}, each with 16MB {{byte[]}}.
> Steps to reproduce:
> # Start Ozone cluster with 1 datanode
> # Start Freon (5K keys of size 1MB)
> Result: OOME after a few hundred keys
> {noformat}
> $ cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone
> $ docker-compose up -d
> $ docker-compose exec scm bash
> $ export HADOOP_OPTS='-XX:+HeapDumpOnOutOfMemoryError'
> $ ozone freon rk --numOfThreads 1 --numOfVolumes 1 --numOfBuckets 1
> --replicationType RATIS --factor ONE --keySize 1048576 --numOfKeys 5120
> --bufferSize 65536
> ...
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid289.hprof ...
> Heap dump file created [1456141975 bytes in 7.760 secs]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2331) Client OOME due to buffer retention
[
https://issues.apache.org/jira/browse/HDDS-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955814#comment-16955814
]
Shashikant Banerjee commented on HDDS-2331:
---
In Ozone, by default the buffer size is equal to the chunk size(16 MB default).
Once a write call happens, a buffer is allocated and data is being written just
into the buffer till it gets full/flush/close and then pushed to datanode and
the buffer gets only released when watchForCommit call for the respective
putBlock call log Index succeeds successfully. So until and unless,
watchForCommit call gets acknowledged by Ratis, we keep holding onto the buffer
so that, in case the ratis request fails, we have the user data cached in the
client buffer which can be written over to the next block.
We have had multiple discussions around this on reducing the default buffer
size and implement a true streaming client, but this is still under
consideration.
[~adoroszlai], for your test, you can try changing the default chunk size to
say 1 MB and see if it works well. It might also be possible that buffer
release handling got broken with some changes introduced which need to be
verified.
> Client OOME due to buffer retention
> ---
>
> Key: HDDS-2331
> URL: https://issues.apache.org/jira/browse/HDDS-2331
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.5.0
>Reporter: Attila Doroszlai
>Assignee: Shashikant Banerjee
>Priority: Critical
> Attachments: profiler.png
>
>
> Freon random key generator exhausts default heap after just few hundred 1MB
> keys. Heap dump on OOME reveals 150+ instances of
> {{ContainerCommandRequestMessage}}, each with 16MB {{byte[]}}.
> Steps to reproduce:
> # Start Ozone cluster with 1 datanode
> # Start Freon (5K keys of size 1MB)
> Result: OOME after a few hundred keys
> {noformat}
> $ cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone
> $ docker-compose up -d
> $ docker-compose exec scm bash
> $ export HADOOP_OPTS='-XX:+HeapDumpOnOutOfMemoryError'
> $ ozone freon rk --numOfThreads 1 --numOfVolumes 1 --numOfBuckets 1
> --replicationType RATIS --factor ONE --keySize 1048576 --numOfKeys 5120
> --bufferSize 65536
> ...
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid289.hprof ...
> Heap dump file created [1456141975 bytes in 7.760 secs]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1574) Ensure same datanodes are not a part of multiple pipelines
[ https://issues.apache.org/jira/browse/HDDS-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955811#comment-16955811 ] Li Cheng commented on HDDS-1574: [~swagle] Cloud you elaborate more to help understand this task? Do you mean we don't want 2 pipelines to share the exactly same group of datanodes as members? > Ensure same datanodes are not a part of multiple pipelines > -- > > Key: HDDS-1574 > URL: https://issues.apache.org/jira/browse/HDDS-1574 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Siddharth Wagle >Assignee: Li Cheng >Priority: Major > > Details in design doc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation
[
https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955804#comment-16955804
]
Xiaoqiao He commented on HDFS-14882:
Thanks [~ayushtkn] for your comments.
{quote}Though things are quiet similar but not for same thing, I think we
should have a separate config.
{quote}
It makes sense to me. [^HDFS-14882.004.patch] add a separate config to govern
this changes. And also extend test case with decommissioned node. Please take
another reviews. Thanks.
> Consider DataNode load when #getBlockLocation
> -
>
> Key: HDFS-14882
> URL: https://issues.apache.org/jira/browse/HDFS-14882
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch,
> HDFS-14882.003.patch, HDFS-14882.004.patch
>
>
> Currently, we consider load of datanode when #chooseTarget for writer,
> however not consider it for reader. Thus, the process slot of datanode could
> be occupied by #BlockSender for reader, and disk/network will be busy
> workload, then meet some slow node exception. IIRC same case is reported
> times. Based on the fact, I propose to consider load for reader same as it
> did #chooseTarget for writer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14916) RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
[
https://issues.apache.org/jira/browse/HDFS-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955802#comment-16955802
]
Ayush Saxena commented on HDFS-14916:
-
Yes. With -d there is a break line, without it, it isn't there for the quota
line. we should add it, for the non -d case.
> RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
> -
>
> Key: HDFS-14916
> URL: https://issues.apache.org/jira/browse/HDFS-14916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf, ui
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Minor
>
> line breaks seem to be missing from the output of "hdfs dfsrouteradmin -ls"
> e.g.:
> The output of "hdfs dfsrouteradmin -ls" now:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1 subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user2
> subCluster1->/tmp/user2user2 users
>rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user/user3
> subCluster1->/user/user3 user3 hadoop
> rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] [sri@nn00070 ~]$
> {code}
> This should be:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls -d
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user2subCluster1->/tmp/user2user2
> users rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user/user3 subCluster1->/user/user3 user3
> hadoop rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> [sri@nn00070 ~]$
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14882) Consider DataNode load when #getBlockLocation
[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-14882: --- Attachment: HDFS-14882.004.patch > Consider DataNode load when #getBlockLocation > - > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch, HDFS-14882.004.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14916) RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
Xieming Li created HDFS-14916:
-
Summary: RBF: line breaks are missing from the output of "hdfs
dfsrouteradmin -ls"
Key: HDFS-14916
URL: https://issues.apache.org/jira/browse/HDFS-14916
Project: Hadoop HDFS
Issue Type: Bug
Components: rbf, ui
Reporter: Xieming Li
Assignee: Xieming Li
line breaks seem to be missing from the output of "hdfs dfsrouteradmin -ls"
e.g.:
The output of "hdfs dfsrouteradmin -ls" now:
{code:java}
[sri@nn00070 ~]$ hdfs dfsrouteradmin -ls
Mount Table Entries:
SourceDestinations Owner
Group Mode Quota/Usage
/testDir1 subCluster1->/user/user1 hdfs
hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user2
subCluster1->/tmp/user2user2 users
rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user/user3
subCluster1->/user/user3 user3 hadoop
rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] [sri@nn00070 ~]$
{code}
This should be:
{code:java}
[sri@nn00070 ~]$ hdfs dfsrouteradmin -ls -d
Mount Table Entries:
SourceDestinations Owner
Group Mode Quota/Usage
/testDir1subCluster1->/user/user1 hdfs
hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
/user2subCluster1->/tmp/user2user2
users rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
/user/user3 subCluster1->/user/user3 user3
hadoop rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
[sri@nn00070 ~]$
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2331) Client OOME due to buffer retention
[
https://issues.apache.org/jira/browse/HDDS-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee reassigned HDDS-2331:
-
Assignee: Shashikant Banerjee
> Client OOME due to buffer retention
> ---
>
> Key: HDDS-2331
> URL: https://issues.apache.org/jira/browse/HDDS-2331
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.5.0
>Reporter: Attila Doroszlai
>Assignee: Shashikant Banerjee
>Priority: Critical
> Attachments: profiler.png
>
>
> Freon random key generator exhausts default heap after just few hundred 1MB
> keys. Heap dump on OOME reveals 150+ instances of
> {{ContainerCommandRequestMessage}}, each with 16MB {{byte[]}}.
> Steps to reproduce:
> # Start Ozone cluster with 1 datanode
> # Start Freon (5K keys of size 1MB)
> Result: OOME after a few hundred keys
> {noformat}
> $ cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone
> $ docker-compose up -d
> $ docker-compose exec scm bash
> $ export HADOOP_OPTS='-XX:+HeapDumpOnOutOfMemoryError'
> $ ozone freon rk --numOfThreads 1 --numOfVolumes 1 --numOfBuckets 1
> --replicationType RATIS --factor ONE --keySize 1048576 --numOfKeys 5120
> --bufferSize 65536
> ...
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid289.hprof ...
> Heap dump file created [1456141975 bytes in 7.760 secs]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
