[jira] [Work logged] (HDDS-2540) Fix accepetance test failure introduced by wait_for_safemode_exit
[
https://issues.apache.org/jira/browse/HDDS-2540?focusedWorklogId=345826=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345826
]
ASF GitHub Bot logged work on HDDS-2540:
Author: ASF GitHub Bot
Created on: 19/Nov/19 06:28
Start Date: 19/Nov/19 06:28
Worklog Time Spent: 10m
Work Description: ChenSammi commented on pull request #222: HDDS-2540.
Fix accepetance test failure introduced by wait_for_safemo…
URL: https://github.com/apache/hadoop-ozone/pull/222
https://issues.apache.org/jira/browse/HDDS-2540
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 345826)
Remaining Estimate: 0h
Time Spent: 10m
> Fix accepetance test failure introduced by wait_for_safemode_exit
> -
>
> Key: HDDS-2540
> URL: https://issues.apache.org/jira/browse/HDDS-2540
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Sammi Chen
>Assignee: Sammi Chen
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> https://github.com/apache/hadoop-ozone/blob/1b72718dcab7f83ebdac67b6242c729f03a8f103/hadoop-ozone/dist/src/main/compose/testlib.sh#L97
> - status=`docker-compose -f "${compose_file}" exec -T scm bash -c
> "kinit -k HTTP/s...@example.com -t /etc/security/keytabs/HTTP.keytab &&
> $command'"`
> + status=`docker-compose -f "${compose_file}" exec -T scm bash -c
> "kinit -k HTTP/s...@example.com -t /etc/security/keytabs/HTTP.keytab &&
> $command"`
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2540) Fix accepetance test failure introduced by wait_for_safemode_exit
[
https://issues.apache.org/jira/browse/HDDS-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-2540:
-
Labels: pull-request-available (was: )
> Fix accepetance test failure introduced by wait_for_safemode_exit
> -
>
> Key: HDDS-2540
> URL: https://issues.apache.org/jira/browse/HDDS-2540
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Sammi Chen
>Assignee: Sammi Chen
>Priority: Major
> Labels: pull-request-available
>
> https://github.com/apache/hadoop-ozone/blob/1b72718dcab7f83ebdac67b6242c729f03a8f103/hadoop-ozone/dist/src/main/compose/testlib.sh#L97
> - status=`docker-compose -f "${compose_file}" exec -T scm bash -c
> "kinit -k HTTP/s...@example.com -t /etc/security/keytabs/HTTP.keytab &&
> $command'"`
> + status=`docker-compose -f "${compose_file}" exec -T scm bash -c
> "kinit -k HTTP/s...@example.com -t /etc/security/keytabs/HTTP.keytab &&
> $command"`
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2540) Fix accepetance test failure introduced by wait_for_safemode_exit
Sammi Chen created HDDS-2540:
Summary: Fix accepetance test failure introduced by
wait_for_safemode_exit
Key: HDDS-2540
URL: https://issues.apache.org/jira/browse/HDDS-2540
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Sammi Chen
Assignee: Sammi Chen
https://github.com/apache/hadoop-ozone/blob/1b72718dcab7f83ebdac67b6242c729f03a8f103/hadoop-ozone/dist/src/main/compose/testlib.sh#L97
- status=`docker-compose -f "${compose_file}" exec -T scm bash -c
"kinit -k HTTP/s...@example.com -t /etc/security/keytabs/HTTP.keytab &&
$command'"`
+ status=`docker-compose -f "${compose_file}" exec -T scm bash -c
"kinit -k HTTP/s...@example.com -t /etc/security/keytabs/HTTP.keytab &&
$command"`
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2486) Sonar: Avoid empty test methods
[
https://issues.apache.org/jira/browse/HDDS-2486?focusedWorklogId=345823=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345823
]
ASF GitHub Bot logged work on HDDS-2486:
Author: ASF GitHub Bot
Created on: 19/Nov/19 06:19
Start Date: 19/Nov/19 06:19
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #220:
HDDS-2486. Sonar: Avoid empty test methods
URL: https://github.com/apache/hadoop-ozone/pull/220
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 345823)
Time Spent: 20m (was: 10m)
> Sonar: Avoid empty test methods
> ---
>
> Key: HDDS-2486
> URL: https://issues.apache.org/jira/browse/HDDS-2486
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Attila Doroszlai
>Assignee: Dinesh Chitlangia
>Priority: Minor
> Labels: pull-request-available, sonar
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{TestRDBTableStore#toIOException}} is empty.
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-5kKcVY8lQ4ZsQH=AW5md-5kKcVY8lQ4ZsQH
> Also {{TestTypedRDBTableStore#toIOException}}:
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-5qKcVY8lQ4ZsQJ=AW5md-5qKcVY8lQ4ZsQJ
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2486) Sonar: Avoid empty test methods
[
https://issues.apache.org/jira/browse/HDDS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-2486:
-
Fix Version/s: 0.5.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Sonar: Avoid empty test methods
> ---
>
> Key: HDDS-2486
> URL: https://issues.apache.org/jira/browse/HDDS-2486
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Attila Doroszlai
>Assignee: Dinesh Chitlangia
>Priority: Minor
> Labels: pull-request-available, sonar
> Fix For: 0.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{TestRDBTableStore#toIOException}} is empty.
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-5kKcVY8lQ4ZsQH=AW5md-5kKcVY8lQ4ZsQH
> Also {{TestTypedRDBTableStore#toIOException}}:
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-5qKcVY8lQ4ZsQJ=AW5md-5qKcVY8lQ4ZsQJ
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7877) [Umbrella] Support maintenance state for datanodes
[
https://issues.apache.org/jira/browse/HDFS-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977142#comment-16977142
]
zy.jordan commented on HDFS-7877:
-
Hi, the maintenance is a great feature. But I have encounter a little problem.
When maintenance time is expired, and some nodes can't get out of maintenance
state. And print some logs.
{quote}2019-11-18 18:45:11,389 INFO [DecommissionMonitor-0] BlockStateChange:
BLOCK* InvalidateBlocks: add blk_52447675210_51376569902 to ip1:50010
2019-11-18 18:45:11,389 INFO [DecommissionMonitor-0]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated 38463
over-replicated blocks on 10.208.51.61:50010 during recommissioning
2019-11-18 18:45:11,389 WARN [DecommissionMonitor-0]
org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager:
DecommissionManager caught exception when processing node.
2019-11-18 18:45:11,389 INFO [DecommissionMonitor-0]
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem write lock
held for 1812 ms via
org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager$Monitor.run(DecommissionManager.java:499)
2019-11-18 18:45:11,389 INFO [DecommissionMonitor-0]
org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager: Checked 0
blocks and 2 nodes this tick
2019-11-18 18:45:24,577 INFO [DecommissionMonitor-0]
org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager: Stopping
maintenance of live node 10.208.51.69:50010
2019-11-18 18:45:24,577 INFO [DecommissionMonitor-0] BlockStateChange: BLOCK*
InvalidateBlocks: add blk_1164445189_90720573 to xip2:50010
{quote}
{quote}2019-11-18 19:02:10,467 WARN [DecommissionMonitor-0]
org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager:
DatanodeAdminMonitor caught exception when processing node xip1:50010.
java.lang.IllegalStateException: Node xip1:50010 is in an invalid
state: In Service Invalid state: 0 blocks are on this dn
at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at
org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager$Monitor.check(DecommissionManager.java:587)
at
org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager$Monitor.run(DecommissionManager.java:495)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{quote}
It shows when it get one node out of maintenance state, it caught +_*exception
when processing node*_+ .
> [Umbrella] Support maintenance state for datanodes
> --
>
> Key: HDFS-7877
> URL: https://issues.apache.org/jira/browse/HDFS-7877
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
>Reporter: Ming Ma
>Assignee: Ming Ma
>Priority: Major
> Fix For: 2.9.0, 3.0.0-beta1, 3.1.0
>
> Attachments: HDFS-7877-2.patch, HDFS-7877.patch,
> Supportmaintenancestatefordatanodes-2.pdf,
> Supportmaintenancestatefordatanodes.pdf
>
>
> This requirement came up during the design for HDFS-7541. Given this feature
> is mostly independent of upgrade domain feature, it is better to track it
> under a separate jira. The design and draft patch will be available soon.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2539) Sonar: Fix synchronization issues in SCMContainerManager class
Siddharth Wagle created HDDS-2539: - Summary: Sonar: Fix synchronization issues in SCMContainerManager class Key: HDDS-2539 URL: https://issues.apache.org/jira/browse/HDDS-2539 Project: Hadoop Distributed Data Store Issue Type: Bug Components: SCM Affects Versions: 0.5.0 Reporter: Siddharth Wagle Assignee: Siddharth Wagle Fix For: 0.5.0 https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-sEKcVY8lQ4ZsDQ=false=BUG -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2538) Sonar: Fix issues found in DatabaseHelper in ozone audit parser package
[ https://issues.apache.org/jira/browse/HDDS-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle updated HDDS-2538: -- Labels: pull-request-available sonar (was: pull-request-available) > Sonar: Fix issues found in DatabaseHelper in ozone audit parser package > --- > > Key: HDDS-2538 > URL: https://issues.apache.org/jira/browse/HDDS-2538 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Tools >Affects Versions: 0.5.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle >Priority: Major > Labels: pull-request-available, sonar > Fix For: 0.5.0 > > Time Spent: 10m > Remaining Estimate: 0h > > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-dWKcVY8lQ4Zr39=false=BLOCKER=BUG -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2538) Sonar: Fix issues found in DatabaseHelper in ozone audit parser package
[ https://issues.apache.org/jira/browse/HDDS-2538?focusedWorklogId=345809=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345809 ] ASF GitHub Bot logged work on HDDS-2538: Author: ASF GitHub Bot Created on: 19/Nov/19 05:49 Start Date: 19/Nov/19 05:49 Worklog Time Spent: 10m Work Description: swagle commented on pull request #221: HDDS-2538. Fix issues found in DatabaseHelper. URL: https://github.com/apache/hadoop-ozone/pull/221 ## What changes were proposed in this pull request? Fixed almost all the sonar warnings in this class, except the cognitive length warning and a strange if condition check which looks totally valid to me; we insert rows if the create table returns true. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2538 ## How was this patch tested? Only syntactic changes made. Waiting on unit tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 345809) Remaining Estimate: 0h Time Spent: 10m > Sonar: Fix issues found in DatabaseHelper in ozone audit parser package > --- > > Key: HDDS-2538 > URL: https://issues.apache.org/jira/browse/HDDS-2538 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Tools >Affects Versions: 0.5.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 10m > Remaining Estimate: 0h > > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-dWKcVY8lQ4Zr39=false=BLOCKER=BUG -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2538) Sonar: Fix issues found in DatabaseHelper in ozone audit parser package
[ https://issues.apache.org/jira/browse/HDDS-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2538: - Labels: pull-request-available (was: ) > Sonar: Fix issues found in DatabaseHelper in ozone audit parser package > --- > > Key: HDDS-2538 > URL: https://issues.apache.org/jira/browse/HDDS-2538 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Tools >Affects Versions: 0.5.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-dWKcVY8lQ4Zr39=false=BLOCKER=BUG -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12866) Recursive delete of a large directory or snapshot makes namenode unresponsive
[
https://issues.apache.org/jira/browse/HDFS-12866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang resolved HDFS-12866.
Resolution: Duplicate
Resolve as it duplicates HDFS-11225
> Recursive delete of a large directory or snapshot makes namenode unresponsive
> -
>
> Key: HDFS-12866
> URL: https://issues.apache.org/jira/browse/HDFS-12866
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Yongjun Zhang
>Priority: Major
>
> Currently file/directory deletion happens in two steps (see
> {{FSNamesystem#delete(String src, boolean recursive, boolean logRetryCache)}}:
> # Do the following under fsn write lock and release the lock afterwards
> ** 1.1 recursively traverse the target, collect INodes and all blocks to be
> deleted
> ** 1.2 delete all INodes
> # Delete the blocks to be deleted incrementally, chunk by chunk. That is, in
> a loop, do:
> ** acquire fsn write lock,
> ** delete chunk of blocks
> ** release fsn write lock
> Breaking the deletion to two steps is to not hold the fsn write lock for too
> long thus making NN not responsive. However, even with this, for deleting
> large directory, or deleting snapshot that has a lot of contents, step 1
> itself would takes long time thus still hold the fsn write lock for too long
> and make NN not responsive.
> A possible solution would be to add one more sub step in step 1, and only
> hold fsn write lock in sub step 1.1:
> * 1.1. hold the fsn write lock, disconnect the target to be deleted from its
> parent dir, release the lock
> * 1.2 recursively traverse the target, collect INodes and all blocks to be
> deleted
> * 1.3 delete all INodes
> Then do step 2.
> This means, any operations on any file/dir need to check if its ancestor is
> deleted (ancestor is disconnected), similar to what's done in
> FSNamesystem#isFileDeleted method.
> I'm throwing the thought here for further discussion. Welcome comments and
> inputs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-11225) NameNode crashed because deleteSnapshot held FSNamesystem lock too long
[
https://issues.apache.org/jira/browse/HDFS-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang resolved HDFS-11225.
Fix Version/s: 3.1.0
Resolution: Fixed
[~shashikant] completed the subtasks. I'm going to resolve this one to avoid
confusion.
For future reference, all the fixes were in 3.1.0 and above.
> NameNode crashed because deleteSnapshot held FSNamesystem lock too long
> ---
>
> Key: HDFS-11225
> URL: https://issues.apache.org/jira/browse/HDFS-11225
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
> Environment: CDH5.8.2, HA
>Reporter: Wei-Chiu Chuang
>Assignee: Shashikant Banerjee
>Priority: Critical
> Labels: high-availability
> Fix For: 3.1.0
>
> Attachments: Snaphot_Deletion_Design_Proposal.pdf
>
>
> The deleteSnapshot operation is synchronous. In certain situations this
> operation may hold FSNamesystem lock for too long, bringing almost every
> NameNode operation to a halt.
> We have observed one incidence where it took so long that ZKFC believes the
> NameNode is down. All other IPC threads were waiting to acquire FSNamesystem
> lock. This specific deleteSnapshot took ~70 seconds. ZKFC has connection
> timeout of 45 seconds by default, and if all IPC threads wait for
> FSNamesystem lock and can't accept new incoming connection, ZKFC times out,
> advances epoch and NameNode will therefore lose its active NN role and then
> fail.
> Relevant log:
> {noformat}
> Thread 154 (IPC Server handler 86 on 8020):
> State: RUNNABLE
> Blocked count: 2753455
> Waited count: 89201773
> Stack:
>
> org.apache.hadoop.hdfs.server.namenode.INode$BlocksMapUpdateInfo.addDeleteBlock(INode.java:879)
>
> org.apache.hadoop.hdfs.server.namenode.INodeFile.destroyAndCollectBlocks(INodeFile.java:508)
>
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.destroyAndCollectBlocks(INodeDirectory.java:763)
>
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.destroyAndCollectBlocks(INodeDirectory.java:763)
>
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.destroyAndCollectBlocks(INodeDirectory.java:763)
>
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.destroyAndCollectBlocks(INodeDirectory.java:763)
>
> org.apache.hadoop.hdfs.server.namenode.INodeReference.destroyAndCollectBlocks(INodeReference.java:339)
>
> org.apache.hadoop.hdfs.server.namenode.INodeReference$WithName.destroyAndCollectBlocks(INodeReference.java:606)
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.DirectoryWithSnapshotFeature$ChildrenDiff.destroyDeletedList(DirectoryWithSnapshotFeature.java:119)
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.DirectoryWithSnapshotFeature$ChildrenDiff.access$400(DirectoryWithSnapshotFeature.java:61)
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.DirectoryWithSnapshotFeature$DirectoryDiff.destroyDiffAndCollectBlocks(DirectoryWithSnapshotFeature.java:319)
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.DirectoryWithSnapshotFeature$DirectoryDiff.destroyDiffAndCollectBlocks(DirectoryWithSnapshotFeature.java:167)
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.AbstractINodeDiffList.deleteSnapshotDiff(AbstractINodeDiffList.java:83)
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.DirectoryWithSnapshotFeature.cleanDirectory(DirectoryWithSnapshotFeature.java:745)
>
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.cleanSubtree(INodeDirectory.java:776)
>
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.cleanSubtreeRecursively(INodeDirectory.java:747)
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.DirectoryWithSnapshotFeature.cleanDirectory(DirectoryWithSnapshotFeature.java:747)
>
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.cleanSubtree(INodeDirectory.java:776)
>
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.cleanSubtreeRecursively(INodeDirectory.java:747)
>
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.cleanSubtree(INodeDirectory.java:789)
> {noformat}
> After the ZKFC determined NameNode was down and advanced epoch, the NN
> finished deleting snapshot, and sent the edit to journal nodes, but it was
> rejected because epoch was updated. See the following stacktrace:
> {noformat}
> 10.0.16.21:8485: IPC's epoch 17 is less than the last promised epoch 18
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.checkRequest(Journal.java:429)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.checkWriteRequest(Journal.java:457)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.journal(Journal.java:352)
> at
>
[jira] [Updated] (HDDS-2486) Sonar: Avoid empty test methods
[
https://issues.apache.org/jira/browse/HDDS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Chitlangia updated HDDS-2486:
Status: Patch Available (was: Open)
> Sonar: Avoid empty test methods
> ---
>
> Key: HDDS-2486
> URL: https://issues.apache.org/jira/browse/HDDS-2486
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Attila Doroszlai
>Assignee: Dinesh Chitlangia
>Priority: Minor
> Labels: pull-request-available, sonar
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{TestRDBTableStore#toIOException}} is empty.
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-5kKcVY8lQ4ZsQH=AW5md-5kKcVY8lQ4ZsQH
> Also {{TestTypedRDBTableStore#toIOException}}:
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-5qKcVY8lQ4ZsQJ=AW5md-5qKcVY8lQ4ZsQJ
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2486) Sonar: Avoid empty test methods
[
https://issues.apache.org/jira/browse/HDDS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-2486:
-
Labels: pull-request-available sonar (was: sonar)
> Sonar: Avoid empty test methods
> ---
>
> Key: HDDS-2486
> URL: https://issues.apache.org/jira/browse/HDDS-2486
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Attila Doroszlai
>Assignee: Dinesh Chitlangia
>Priority: Minor
> Labels: pull-request-available, sonar
>
> {{TestRDBTableStore#toIOException}} is empty.
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-5kKcVY8lQ4ZsQH=AW5md-5kKcVY8lQ4ZsQH
> Also {{TestTypedRDBTableStore#toIOException}}:
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-5qKcVY8lQ4ZsQJ=AW5md-5qKcVY8lQ4ZsQJ
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2486) Sonar: Avoid empty test methods
[
https://issues.apache.org/jira/browse/HDDS-2486?focusedWorklogId=345806=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345806
]
ASF GitHub Bot logged work on HDDS-2486:
Author: ASF GitHub Bot
Created on: 19/Nov/19 05:29
Start Date: 19/Nov/19 05:29
Worklog Time Spent: 10m
Work Description: dineshchitlangia commented on pull request #220:
HDDS-2486. Sonar: Avoid empty test methods
URL: https://github.com/apache/hadoop-ozone/pull/220
## What changes were proposed in this pull request?
Removed empty test methods `TestRDBTableStore#toIOException` and
`TestTypedRDBTableStore#toIOException`
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-2486
## How was this patch tested?
N/A
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 345806)
Remaining Estimate: 0h
Time Spent: 10m
> Sonar: Avoid empty test methods
> ---
>
> Key: HDDS-2486
> URL: https://issues.apache.org/jira/browse/HDDS-2486
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Attila Doroszlai
>Assignee: Dinesh Chitlangia
>Priority: Minor
> Labels: pull-request-available, sonar
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{TestRDBTableStore#toIOException}} is empty.
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-5kKcVY8lQ4ZsQH=AW5md-5kKcVY8lQ4ZsQH
> Also {{TestTypedRDBTableStore#toIOException}}:
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-5qKcVY8lQ4ZsQJ=AW5md-5qKcVY8lQ4ZsQJ
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2486) Sonar: Avoid empty test methods
[
https://issues.apache.org/jira/browse/HDDS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Chitlangia updated HDDS-2486:
Summary: Sonar: Avoid empty test methods (was: Empty test method in
TestRDBTableStore)
> Sonar: Avoid empty test methods
> ---
>
> Key: HDDS-2486
> URL: https://issues.apache.org/jira/browse/HDDS-2486
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Attila Doroszlai
>Assignee: Dinesh Chitlangia
>Priority: Minor
> Labels: sonar
>
> {{TestRDBTableStore#toIOException}} is empty.
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-5kKcVY8lQ4ZsQH=AW5md-5kKcVY8lQ4ZsQH
> Also {{TestTypedRDBTableStore#toIOException}}:
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-5qKcVY8lQ4ZsQJ=AW5md-5qKcVY8lQ4ZsQJ
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2486) Empty test method in TestRDBTableStore
[
https://issues.apache.org/jira/browse/HDDS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Chitlangia reassigned HDDS-2486:
---
Assignee: Dinesh Chitlangia
> Empty test method in TestRDBTableStore
> --
>
> Key: HDDS-2486
> URL: https://issues.apache.org/jira/browse/HDDS-2486
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Attila Doroszlai
>Assignee: Dinesh Chitlangia
>Priority: Minor
> Labels: sonar
>
> {{TestRDBTableStore#toIOException}} is empty.
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-5kKcVY8lQ4ZsQH=AW5md-5kKcVY8lQ4ZsQH
> Also {{TestTypedRDBTableStore#toIOException}}:
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-5qKcVY8lQ4ZsQJ=AW5md-5qKcVY8lQ4ZsQJ
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2538) Sonar: Fix issues found in DatabaseHelper in ozone audit parser package
[ https://issues.apache.org/jira/browse/HDDS-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle updated HDDS-2538: -- Summary: Sonar: Fix issues found in DatabaseHelper in ozone audit parser package (was: Sonar: Fix issues found in DatabaseHelper in ozone audit parser) > Sonar: Fix issues found in DatabaseHelper in ozone audit parser package > --- > > Key: HDDS-2538 > URL: https://issues.apache.org/jira/browse/HDDS-2538 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Tools >Affects Versions: 0.5.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle >Priority: Major > Fix For: 0.5.0 > > > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-dWKcVY8lQ4Zr39=false=BLOCKER=BUG -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2538) Sonar: Fix issues found in DatabaseHelper in ozone audit parser
Siddharth Wagle created HDDS-2538: - Summary: Sonar: Fix issues found in DatabaseHelper in ozone audit parser Key: HDDS-2538 URL: https://issues.apache.org/jira/browse/HDDS-2538 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Tools Affects Versions: 0.5.0 Reporter: Siddharth Wagle Assignee: Siddharth Wagle Fix For: 0.5.0 https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-dWKcVY8lQ4Zr39=false=BLOCKER=BUG -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2514) Remove this unused method parameter "encodedToken"
[ https://issues.apache.org/jira/browse/HDDS-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia resolved HDDS-2514. - Fix Version/s: 0.5.0 Target Version/s: 0.5.0 Resolution: Fixed Thanks [~apurohit]for the contribution and [~bharat] for the reviews. > Remove this unused method parameter "encodedToken" > -- > > Key: HDDS-2514 > URL: https://issues.apache.org/jira/browse/HDDS-2514 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Abhishek Purohit >Assignee: Abhishek Purohit >Priority: Major > Labels: pull-request-available, sonar > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Remove this unused method parameter "encodedToken" > method: connectToDatanode > Class: XceiverClientGrpc > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2514) Remove this unused method parameter "encodedToken"
[ https://issues.apache.org/jira/browse/HDDS-2514?focusedWorklogId=345800=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345800 ] ASF GitHub Bot logged work on HDDS-2514: Author: ASF GitHub Bot Created on: 19/Nov/19 04:57 Start Date: 19/Nov/19 04:57 Worklog Time Spent: 10m Work Description: dineshchitlangia commented on pull request #189: HDDS-2514. removed unused method param URL: https://github.com/apache/hadoop-ozone/pull/189 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 345800) Time Spent: 20m (was: 10m) > Remove this unused method parameter "encodedToken" > -- > > Key: HDDS-2514 > URL: https://issues.apache.org/jira/browse/HDDS-2514 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Abhishek Purohit >Assignee: Abhishek Purohit >Priority: Major > Labels: pull-request-available, sonar > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Remove this unused method parameter "encodedToken" > method: connectToDatanode > Class: XceiverClientGrpc > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2517) Immediately return this expression instead of assigning it to the temporary variable
[ https://issues.apache.org/jira/browse/HDDS-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia resolved HDDS-2517. - Fix Version/s: 0.5.0 Target Version/s: 0.5.0 Resolution: Fixed [~apurohit] Thanks for filing the issue and providing a fix. [~adoroszlai] & [~bharat] for the reviews. > Immediately return this expression instead of assigning it to the temporary > variable > > > Key: HDDS-2517 > URL: https://issues.apache.org/jira/browse/HDDS-2517 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Abhishek Purohit >Assignee: Abhishek Purohit >Priority: Major > Labels: pull-request-available, sonar > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Related to : > [https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md_AGKcVY8lQ4ZsV1=false] > Immediately return this expression instead of assigning it to the temporary > variable -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2517) Immediately return this expression instead of assigning it to the temporary variable
[ https://issues.apache.org/jira/browse/HDDS-2517?focusedWorklogId=345799=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345799 ] ASF GitHub Bot logged work on HDDS-2517: Author: ASF GitHub Bot Created on: 19/Nov/19 04:53 Start Date: 19/Nov/19 04:53 Worklog Time Spent: 10m Work Description: dineshchitlangia commented on pull request #192: HDDS-2517. Immediately return rather than holding to variable and the… URL: https://github.com/apache/hadoop-ozone/pull/192 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 345799) Time Spent: 20m (was: 10m) > Immediately return this expression instead of assigning it to the temporary > variable > > > Key: HDDS-2517 > URL: https://issues.apache.org/jira/browse/HDDS-2517 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Abhishek Purohit >Assignee: Abhishek Purohit >Priority: Major > Labels: pull-request-available, sonar > Time Spent: 20m > Remaining Estimate: 0h > > Related to : > [https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md_AGKcVY8lQ4ZsV1=false] > Immediately return this expression instead of assigning it to the temporary > variable -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2527) Sonar : remove redundant temporary assignment in HddsVersionProvider
[ https://issues.apache.org/jira/browse/HDDS-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia resolved HDDS-2527. - Fix Version/s: 0.5.0 Target Version/s: 0.5.0 Resolution: Fixed [~sdeka] thanks for reporting the issue, [~shwetayakkali] thanks for the contribution. > Sonar : remove redundant temporary assignment in HddsVersionProvider > > > Key: HDDS-2527 > URL: https://issues.apache.org/jira/browse/HDDS-2527 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Supratim Deka >Assignee: Shweta >Priority: Minor > Labels: pull-request-available, sonar > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Sonar report : > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-4AKcVY8lQ4ZsO6=AW5md-4AKcVY8lQ4ZsO6 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2527) Sonar : remove redundant temporary assignment in HddsVersionProvider
[ https://issues.apache.org/jira/browse/HDDS-2527?focusedWorklogId=345795=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345795 ] ASF GitHub Bot logged work on HDDS-2527: Author: ASF GitHub Bot Created on: 19/Nov/19 04:36 Start Date: 19/Nov/19 04:36 Worklog Time Spent: 10m Work Description: dineshchitlangia commented on pull request #219: HDDS-2527. Sonar : remove redundant temporary assignment in HddsVersionProvider URL: https://github.com/apache/hadoop-ozone/pull/219 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 345795) Time Spent: 20m (was: 10m) > Sonar : remove redundant temporary assignment in HddsVersionProvider > > > Key: HDDS-2527 > URL: https://issues.apache.org/jira/browse/HDDS-2527 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Supratim Deka >Assignee: Shweta >Priority: Minor > Labels: pull-request-available, sonar > Time Spent: 20m > Remaining Estimate: 0h > > Sonar report : > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-4AKcVY8lQ4ZsO6=AW5md-4AKcVY8lQ4ZsO6 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2490) Ensure OzoneClient is closed in Ozone Shell handlers
[
https://issues.apache.org/jira/browse/HDDS-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Chitlangia resolved HDDS-2490.
-
Fix Version/s: 0.5.0
Resolution: Fixed
[~adoroszlai] Thanks for filing the issue, [~ppogde] thanks for the
contribution.
> Ensure OzoneClient is closed in Ozone Shell handlers
>
>
> Key: HDDS-2490
> URL: https://issues.apache.org/jira/browse/HDDS-2490
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Reporter: Attila Doroszlai
>Assignee: Prashant Pogde
>Priority: Minor
> Labels: pull-request-available, sonar
> Fix For: 0.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> OzoneClient should be closed in all command handlers ({{Handler}} subclasses).
> *
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-Y0KcVY8lQ4Zrz6=AW5md-Y0KcVY8lQ4Zrz6
> *
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-ZGKcVY8lQ4Zr0b=AW5md-ZGKcVY8lQ4Zr0b
> etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2490) Ensure OzoneClient is closed in Ozone Shell handlers
[
https://issues.apache.org/jira/browse/HDDS-2490?focusedWorklogId=345791=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345791
]
ASF GitHub Bot logged work on HDDS-2490:
Author: ASF GitHub Bot
Created on: 19/Nov/19 04:33
Start Date: 19/Nov/19 04:33
Worklog Time Spent: 10m
Work Description: dineshchitlangia commented on pull request #217:
HDDS-2490. Fixing sonarcloud errors.
URL: https://github.com/apache/hadoop-ozone/pull/217
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 345791)
Time Spent: 20m (was: 10m)
> Ensure OzoneClient is closed in Ozone Shell handlers
>
>
> Key: HDDS-2490
> URL: https://issues.apache.org/jira/browse/HDDS-2490
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Reporter: Attila Doroszlai
>Assignee: Prashant Pogde
>Priority: Minor
> Labels: pull-request-available, sonar
> Time Spent: 20m
> Remaining Estimate: 0h
>
> OzoneClient should be closed in all command handlers ({{Handler}} subclasses).
> *
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-Y0KcVY8lQ4Zrz6=AW5md-Y0KcVY8lQ4Zrz6
> *
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-ZGKcVY8lQ4Zr0b=AW5md-ZGKcVY8lQ4Zr0b
> etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14991) Backport better time precision of configuration#getTimeDuration to branch-2 to support SBN read
[ https://issues.apache.org/jira/browse/HDFS-14991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-14991: -- Fix Version/s: 2.10.1 > Backport better time precision of configuration#getTimeDuration to branch-2 > to support SBN read > --- > > Key: HDFS-14991 > URL: https://issues.apache.org/jira/browse/HDFS-14991 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Fix For: 2.10.1 > > Attachments: HDFS-14991-branch-2.001.patch, > HDFS-14991-branch-2.002.patch > > > This Jira targets to backport changes in HDFS-14346 and HADOOP-16265 to allow > better time precision of configuration#getTimeDuration. This is required by > Standby read, where edit log tailing period needs to be set small. > Note that HDFS-14346 is only partially backported in this Jira, as it has > changes based on previous change from HDFS-9847. HDFS-9847 backport is not in > the scope of this Jira. Therefore, this Jira only targets at backporting the > minimum change required by Standby read. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14991) Backport better time precision of configuration#getTimeDuration to branch-2 to support SBN read
[ https://issues.apache.org/jira/browse/HDFS-14991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-14991: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to branch-2, thanks Erik and Chao for the reviews! > Backport better time precision of configuration#getTimeDuration to branch-2 > to support SBN read > --- > > Key: HDFS-14991 > URL: https://issues.apache.org/jira/browse/HDFS-14991 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-14991-branch-2.001.patch, > HDFS-14991-branch-2.002.patch > > > This Jira targets to backport changes in HDFS-14346 and HADOOP-16265 to allow > better time precision of configuration#getTimeDuration. This is required by > Standby read, where edit log tailing period needs to be set small. > Note that HDFS-14346 is only partially backported in this Jira, as it has > changes based on previous change from HDFS-9847. HDFS-9847 backport is not in > the scope of this Jira. Therefore, this Jira only targets at backporting the > minimum change required by Standby read. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-426) Add field modificationTime for Volume and Bucket
[
https://issues.apache.org/jira/browse/HDDS-426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977108#comment-16977108
]
Dinesh Chitlangia commented on HDDS-426:
Theoretically, if we perform any key related operation, we are modifying the
related bucket and volume.
Based on this, I think we should update modificationTime for Volume & Bucket
whenever we:
- Create Key
- Delete Key
- Modify Key
[~aengineer] , [~arp] - I am concerned that this will be too much of an
overhead. Your thoughts?
> Add field modificationTime for Volume and Bucket
>
>
> Key: HDDS-426
> URL: https://issues.apache.org/jira/browse/HDDS-426
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Manager
>Reporter: Dinesh Chitlangia
>Assignee: YiSheng Lien
>Priority: Major
> Labels: newbie, pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There are update operations that can be performed for Volume, Bucket and Key.
> While Key records the modification time, Volume and & Bucket do not capture
> this.
>
> This Jira proposes to add the required field to Volume and Bucket in order to
> capture the modficationTime.
>
> Current Status:
> {noformat}
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoVolume /dummyvol
> 2018-09-10 17:16:12 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "owner" : {
> "name" : "bilbo"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "dummyvol",
> "createdOn" : "Mon, 10 Sep 2018 17:11:32 GMT",
> "createdBy" : "bilbo"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoBucket /dummyvol/mybuck
> 2018-09-10 17:15:25 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "volumeName" : "dummyvol",
> "bucketName" : "mybuck",
> "createdOn" : "Mon, 10 Sep 2018 17:12:09 GMT",
> "acls" : [ {
> "type" : "USER",
> "name" : "hadoop",
> "rights" : "READ_WRITE"
> }, {
> "type" : "GROUP",
> "name" : "users",
> "rights" : "READ_WRITE"
> }, {
> "type" : "USER",
> "name" : "spark",
> "rights" : "READ_WRITE"
> } ],
> "versioning" : "DISABLED",
> "storageType" : "DISK"
> }
> hadoop@1987b5de4203:~$ ./bin/ozone oz -infoKey /dummyvol/mybuck/myk1
> 2018-09-10 17:19:43 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "modifiedOn" : "Mon, 10 Sep 2018 17:19:04 GMT",
> "size" : 0,
> "keyName" : "myk1",
> "keyLocations" : [ ]
> }{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2499) IsLeader information is lost when update pipeline state
[ https://issues.apache.org/jira/browse/HDDS-2499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen resolved HDDS-2499. -- Fix Version/s: 0.5.0 Resolution: Fixed > IsLeader information is lost when update pipeline state > --- > > Key: HDDS-2499 > URL: https://issues.apache.org/jira/browse/HDDS-2499 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1572) Implement a Pipeline scrubber to maintain healthy number of pipelines in a cluster
[ https://issues.apache.org/jira/browse/HDDS-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-1572: --- Description: The design document talks about initial requirements for the pipeline scrubber. - Scan the pipelines that the nodes are a part of to select candidates for teardown. - Scan pipelines that do not have open containers currently in use and datanodes are in violation. - Schedule tear down operation if a candidate pipeline is found. was: The design document talks about initial requirements for the pipeline scrubber. - Maintain a datastructure for datanodes violating the pipeline membership soft upper bound. - Scan the pipelines that the nodes are a part of to select candidates for teardown. - Scan pipelines that do not have open containers currently in use and datanodes are in violation. - Schedule tear down operation if a candidate pipeline is found. > Implement a Pipeline scrubber to maintain healthy number of pipelines in a > cluster > -- > > Key: HDDS-1572 > URL: https://issues.apache.org/jira/browse/HDDS-1572 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Siddharth Wagle >Assignee: Li Cheng >Priority: Major > > The design document talks about initial requirements for the pipeline > scrubber. > - Scan the pipelines that the nodes are a part of to select candidates for > teardown. > - Scan pipelines that do not have open containers currently in use and > datanodes are in violation. > - Schedule tear down operation if a candidate pipeline is found. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2527) Sonar : remove redundant temporary assignment in HddsVersionProvider
[ https://issues.apache.org/jira/browse/HDDS-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2527: - Labels: pull-request-available sonar (was: sonar) > Sonar : remove redundant temporary assignment in HddsVersionProvider > > > Key: HDDS-2527 > URL: https://issues.apache.org/jira/browse/HDDS-2527 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Supratim Deka >Assignee: Shweta >Priority: Minor > Labels: pull-request-available, sonar > > Sonar report : > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-4AKcVY8lQ4ZsO6=AW5md-4AKcVY8lQ4ZsO6 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2527) Sonar : remove redundant temporary assignment in HddsVersionProvider
[ https://issues.apache.org/jira/browse/HDDS-2527?focusedWorklogId=345768=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345768 ] ASF GitHub Bot logged work on HDDS-2527: Author: ASF GitHub Bot Created on: 19/Nov/19 02:22 Start Date: 19/Nov/19 02:22 Worklog Time Spent: 10m Work Description: shwetayakkali commented on pull request #219: HDDS-2527. Sonar : remove redundant temporary assignment in HddsVersionProvider URL: https://github.com/apache/hadoop-ozone/pull/219 ## What changes were proposed in this pull request? This Jira removes the use of redundant temporary String array 'result'. Sonar report : https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-4AKcVY8lQ4ZsO6=AW5md-4AKcVY8lQ4ZsO6 ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2527 ## How was this patch tested? N/A This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 345768) Remaining Estimate: 0h Time Spent: 10m > Sonar : remove redundant temporary assignment in HddsVersionProvider > > > Key: HDDS-2527 > URL: https://issues.apache.org/jira/browse/HDDS-2527 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Supratim Deka >Assignee: Shweta >Priority: Minor > Labels: pull-request-available, sonar > Time Spent: 10m > Remaining Estimate: 0h > > Sonar report : > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-4AKcVY8lQ4ZsO6=AW5md-4AKcVY8lQ4ZsO6 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2499) IsLeader information is lost when update pipeline state
[ https://issues.apache.org/jira/browse/HDDS-2499?focusedWorklogId=345767=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345767 ] ASF GitHub Bot logged work on HDDS-2499: Author: ASF GitHub Bot Created on: 19/Nov/19 02:21 Start Date: 19/Nov/19 02:21 Worklog Time Spent: 10m Work Description: ChenSammi commented on pull request #180: HDDS-2499. IsLeader information is lost when update pipeline state. URL: https://github.com/apache/hadoop-ozone/pull/180 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 345767) Time Spent: 20m (was: 10m) > IsLeader information is lost when update pipeline state > --- > > Key: HDDS-2499 > URL: https://issues.apache.org/jira/browse/HDDS-2499 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14991) Backport better time precision of configuration#getTimeDuration to branch-2 to support SBN read
[ https://issues.apache.org/jira/browse/HDFS-14991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977022#comment-16977022 ] Chen Liang commented on HDFS-14991: --- Thanks Erik! Updated description and title. Will commit shortly. > Backport better time precision of configuration#getTimeDuration to branch-2 > to support SBN read > --- > > Key: HDFS-14991 > URL: https://issues.apache.org/jira/browse/HDFS-14991 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-14991-branch-2.001.patch, > HDFS-14991-branch-2.002.patch > > > This Jira targets to backport changes in HDFS-14346 and HADOOP-16265 to allow > better time precision of configuration#getTimeDuration. This is required by > Standby read, where edit log tailing period needs to be set small. > Note that HDFS-14346 is only partially backported in this Jira, as it has > changes based on previous change from HDFS-9847. HDFS-9847 backport is not in > the scope of this Jira. Therefore, this Jira only targets at backporting the > minimum change required by Standby read. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14991) Backport better time precision of configuration#getTimeDuration to branch-2 to support SBN read
[ https://issues.apache.org/jira/browse/HDFS-14991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-14991: -- Summary: Backport better time precision of configuration#getTimeDuration to branch-2 to support SBN read (was: Backport HDFS-14346 Better time precision in getTimeDuration to branch-2) > Backport better time precision of configuration#getTimeDuration to branch-2 > to support SBN read > --- > > Key: HDFS-14991 > URL: https://issues.apache.org/jira/browse/HDFS-14991 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-14991-branch-2.001.patch, > HDFS-14991-branch-2.002.patch > > > This is to backport HDFS-14346 to branch 2, as Standby reads in branch-2 > requires being able to properly specify ms time granularity for Edit log > tailing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14991) Backport better time precision of configuration#getTimeDuration to branch-2 to support SBN read
[ https://issues.apache.org/jira/browse/HDFS-14991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-14991: -- Description: This Jira targets to backport changes in HDFS-14346 and HADOOP-16265 to allow better time precision of configuration#getTimeDuration. This is required by Standby read, where edit log tailing period needs to be set small. Note that HDFS-14346 is only partially backported in this Jira, as it has changes based on previous change from HDFS-9847. HDFS-9847 backport is not in the scope of this Jira. Therefore, this Jira only targets at backporting the minimum change required by Standby read. was:This is to backport HDFS-14346 to branch 2, as Standby reads in branch-2 requires being able to properly specify ms time granularity for Edit log tailing. > Backport better time precision of configuration#getTimeDuration to branch-2 > to support SBN read > --- > > Key: HDFS-14991 > URL: https://issues.apache.org/jira/browse/HDFS-14991 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-14991-branch-2.001.patch, > HDFS-14991-branch-2.002.patch > > > This Jira targets to backport changes in HDFS-14346 and HADOOP-16265 to allow > better time precision of configuration#getTimeDuration. This is required by > Standby read, where edit log tailing period needs to be set small. > Note that HDFS-14346 is only partially backported in this Jira, as it has > changes based on previous change from HDFS-9847. HDFS-9847 backport is not in > the scope of this Jira. Therefore, this Jira only targets at backporting the > minimum change required by Standby read. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2531) Sonar : remove duplicate string literals in BlockOutputStream
[ https://issues.apache.org/jira/browse/HDDS-2531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shweta reassigned HDDS-2531: Assignee: Shweta (was: Supratim Deka) > Sonar : remove duplicate string literals in BlockOutputStream > - > > Key: HDDS-2531 > URL: https://issues.apache.org/jira/browse/HDDS-2531 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Reporter: Supratim Deka >Assignee: Shweta >Priority: Minor > Labels: sonar > > Sonar issue in executePutBlock, duplicate string literal "blockID" : > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-_1KcVY8lQ4ZsVa=AW5md-_1KcVY8lQ4ZsVa > format specifiers in Log: > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-_2KcVY8lQ4ZsVg=AW5md-_2KcVY8lQ4ZsVg > define string constant instead of duplicate string literals. > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-_2KcVY8lQ4ZsVb=AW5md-_2KcVY8lQ4ZsVb -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2527) Sonar : remove redundant temporary assignment in HddsVersionProvider
[ https://issues.apache.org/jira/browse/HDDS-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shweta reassigned HDDS-2527: Assignee: Shweta (was: Supratim Deka) > Sonar : remove redundant temporary assignment in HddsVersionProvider > > > Key: HDDS-2527 > URL: https://issues.apache.org/jira/browse/HDDS-2527 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Supratim Deka >Assignee: Shweta >Priority: Minor > Labels: sonar > > Sonar report : > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-4AKcVY8lQ4ZsO6=AW5md-4AKcVY8lQ4ZsO6 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14991) Backport HDFS-14346 Better time precision in getTimeDuration to branch-2
[ https://issues.apache.org/jira/browse/HDFS-14991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976991#comment-16976991 ] Erik Krogen commented on HDFS-14991: I don't think we'll backport HDFS-9847 -- as noted in some of the comments near the end of that JIRA conversation, in its current form it's not really a compatible backport as it changes some of the default values. I think it's probably okay to include the change here, but agreed that we should make both the commit message and JIRA description very clearly reflect exactly what was committed. +1 from me assuming you make the checkstyle fix and the description updates. > Backport HDFS-14346 Better time precision in getTimeDuration to branch-2 > > > Key: HDFS-14991 > URL: https://issues.apache.org/jira/browse/HDFS-14991 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-14991-branch-2.001.patch, > HDFS-14991-branch-2.002.patch > > > This is to backport HDFS-14346 to branch 2, as Standby reads in branch-2 > requires being able to properly specify ms time granularity for Edit log > tailing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14991) Backport HDFS-14346 Better time precision in getTimeDuration to branch-2
[ https://issues.apache.org/jira/browse/HDFS-14991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976984#comment-16976984 ] Chen Liang commented on HDFS-14991: --- The checkstyle warning is a line len > 80 of log print line. Which I consider does not worth another patch submit, I can fix when commit, if no other issue on v02 patch. > Backport HDFS-14346 Better time precision in getTimeDuration to branch-2 > > > Key: HDFS-14991 > URL: https://issues.apache.org/jira/browse/HDFS-14991 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-14991-branch-2.001.patch, > HDFS-14991-branch-2.002.patch > > > This is to backport HDFS-14346 to branch 2, as Standby reads in branch-2 > requires being able to properly specify ms time granularity for Edit log > tailing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14991) Backport HDFS-14346 Better time precision in getTimeDuration to branch-2
[ https://issues.apache.org/jira/browse/HDFS-14991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976976#comment-16976976 ] Chen Liang commented on HDFS-14991: --- Thanks Chao!! I think I still need to get a binding +1, ping [~xkrogen] and [~shv], can you please take a look? The failed tests all passed in my local run. > Backport HDFS-14346 Better time precision in getTimeDuration to branch-2 > > > Key: HDFS-14991 > URL: https://issues.apache.org/jira/browse/HDFS-14991 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-14991-branch-2.001.patch, > HDFS-14991-branch-2.002.patch > > > This is to backport HDFS-14346 to branch 2, as Standby reads in branch-2 > requires being able to properly specify ms time granularity for Edit log > tailing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7784) load fsimage in parallel
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-7784: -- Target Version/s: (was: ) Resolution: Duplicate Status: Resolved (was: Patch Available) I'm going to resolve this one since HDFS-14617 took large part of the code from here and merged in the codebase. > load fsimage in parallel > > > Key: HDFS-7784 > URL: https://issues.apache.org/jira/browse/HDFS-7784 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Walter Su >Assignee: Walter Su >Priority: Minor > Labels: BB2015-05-TBR > Attachments: HDFS-7784.001.patch, test-20150213.pdf > > > When single Namenode has huge amount of files, without using federation, the > startup/restart speed is slow. The fsimage loading step takes the most of the > time. fsimage loading can seperate to two parts, deserialization and object > construction(mostly map insertion). Deserialization takes the most of CPU > time. So we can do deserialization in parallel, and add to hashmap in serial. > It will significantly reduce the NN start time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2536) Add ozone.om.internal.service.id to OM HA configuration
[ https://issues.apache.org/jira/browse/HDDS-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-2536: - Parent: HDDS-505 Issue Type: Sub-task (was: Bug) > Add ozone.om.internal.service.id to OM HA configuration > --- > > Key: HDDS-2536 > URL: https://issues.apache.org/jira/browse/HDDS-2536 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to add ozone.om.internal.serviceid to let OM knows it belong to > a particular service. > > As now we have ozone.om.service.ids -≥ where we can define all service id's > in a cluster.(This can happen if the same config is shared across the cluster) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2536) Add ozone.om.internal.service.id to OM HA configuration
[ https://issues.apache.org/jira/browse/HDDS-2536?focusedWorklogId=345636=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345636 ] ASF GitHub Bot logged work on HDDS-2536: Author: ASF GitHub Bot Created on: 18/Nov/19 22:32 Start Date: 18/Nov/19 22:32 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #218: HDDS-2536. Add ozone.om.internal.service.id to OM HA configuration. URL: https://github.com/apache/hadoop-ozone/pull/218 ## What changes were proposed in this pull request? This Jira is to add ozone.om.internal.serviceid to let OM knows it belongs to a particular service. As now we have ozone.om.service.ids -≥ where we can define all service id's in a cluster.(This can happen if the same config is shared across the cluster) ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2536 ## How was this patch tested? Modified MiniOzoneHAclusterImpl to set serviceID value for ozone.om.internal.service.id. And ran TestOzoneFSHAUrls.java, and test passed. And also verified that the new code path is being run. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 345636) Remaining Estimate: 0h Time Spent: 10m > Add ozone.om.internal.service.id to OM HA configuration > --- > > Key: HDDS-2536 > URL: https://issues.apache.org/jira/browse/HDDS-2536 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to add ozone.om.internal.serviceid to let OM knows it belong to > a particular service. > > As now we have ozone.om.service.ids -≥ where we can define all service id's > in a cluster.(This can happen if the same config is shared across the cluster) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2536) Add ozone.om.internal.service.id to OM HA configuration
[ https://issues.apache.org/jira/browse/HDDS-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2536: - Labels: pull-request-available (was: ) > Add ozone.om.internal.service.id to OM HA configuration > --- > > Key: HDDS-2536 > URL: https://issues.apache.org/jira/browse/HDDS-2536 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > > This Jira is to add ozone.om.internal.serviceid to let OM knows it belong to > a particular service. > > As now we have ozone.om.service.ids -≥ where we can define all service id's > in a cluster.(This can happen if the same config is shared across the cluster) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2537) Opentracing - Deprecated Members
Matthew Sharp created HDDS-2537: --- Summary: Opentracing - Deprecated Members Key: HDDS-2537 URL: https://issues.apache.org/jira/browse/HDDS-2537 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: Matthew Sharp When the opentracing-util dependency is upgraded, there will be future cleanup work needed with a couple deprecated members starting with 0.31.0 (current version used today). A full deprecated members list can be found here: [https://github.com/opentracing/opentracing-java] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14973) Balancer getBlocks RPC dispersal does not function properly
[
https://issues.apache.org/jira/browse/HDFS-14973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976939#comment-16976939
]
Hadoop QA commented on HDFS-14973:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
33s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 644 unchanged - 1 fixed = 645 total (was 645) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 21s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}113m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:f555aa740b5 |
| JIRA Issue | HDFS-14973 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12986154/HDFS-14973-branch-2.004.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 5837485df68e 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64
[jira] [Created] (HDDS-2536) Add ozone.om.internal.service.id to OM HA configuration
Bharat Viswanadham created HDDS-2536: Summary: Add ozone.om.internal.service.id to OM HA configuration Key: HDDS-2536 URL: https://issues.apache.org/jira/browse/HDDS-2536 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham This Jira is to add ozone.om.internal.serviceid to let OM knows it belong to a particular service. As now we have ozone.om.service.ids -≥ where we can define all service id's in a cluster.(This can happen if the same config is shared across the cluster) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1987) Fix listStatus API
[ https://issues.apache.org/jira/browse/HDDS-1987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDDS-1987: - Status: Patch Available (was: In Progress) > Fix listStatus API > -- > > Key: HDDS-1987 > URL: https://issues.apache.org/jira/browse/HDDS-1987 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to fix listStatus API in HA code path. > In HA, we have an in-memory cache, where we put the result to in-memory cache > and return the response. It will be picked by double buffer thread and > flushed to disk later. So when user call listStatus, it should use both > in-memory cache and rocksdb key table to return the correct result. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2455) Implement MiniOzoneHAClusterImpl#getOMLeader
[ https://issues.apache.org/jira/browse/HDDS-2455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDDS-2455: - Status: Patch Available (was: In Progress) > Implement MiniOzoneHAClusterImpl#getOMLeader > > > Key: HDDS-2455 > URL: https://issues.apache.org/jira/browse/HDDS-2455 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Implement MiniOzoneHAClusterImpl#getOMLeader and use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2105) Merge OzoneClientFactory#getRpcClient functions
[ https://issues.apache.org/jira/browse/HDDS-2105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDDS-2105: - Status: Patch Available (was: In Progress) > Merge OzoneClientFactory#getRpcClient functions > --- > > Key: HDDS-2105 > URL: https://issues.apache.org/jira/browse/HDDS-2105 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Ref: https://github.com/apache/hadoop/pull/1360#discussion_r321585214 > There will be 5 overloaded OzoneClientFactory#getRpcClient functions (when > HDDS-2007 is committed). They contains some redundant logic and unnecessarily > increases code paths. > Goal: Merge those functions into fewer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2490) Ensure OzoneClient is closed in Ozone Shell handlers
[
https://issues.apache.org/jira/browse/HDDS-2490?focusedWorklogId=345609=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345609
]
ASF GitHub Bot logged work on HDDS-2490:
Author: ASF GitHub Bot
Created on: 18/Nov/19 21:57
Start Date: 18/Nov/19 21:57
Worklog Time Spent: 10m
Work Description: prashantpogde commented on pull request #217:
HDDS-2490. Fixing sonarcloud errors.
URL: https://github.com/apache/hadoop-ozone/pull/217
## What changes were proposed in this pull request?
Fixing sonar cloud Warning: Ensure OzoneClient is closed in Ozone Shell
handlers
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-2490
## How was this patch tested?
Clean Build and basic smoketest.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 345609)
Remaining Estimate: 0h
Time Spent: 10m
> Ensure OzoneClient is closed in Ozone Shell handlers
>
>
> Key: HDDS-2490
> URL: https://issues.apache.org/jira/browse/HDDS-2490
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Reporter: Attila Doroszlai
>Assignee: Prashant Pogde
>Priority: Minor
> Labels: pull-request-available, sonar
> Time Spent: 10m
> Remaining Estimate: 0h
>
> OzoneClient should be closed in all command handlers ({{Handler}} subclasses).
> *
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-Y0KcVY8lQ4Zrz6=AW5md-Y0KcVY8lQ4Zrz6
> *
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-ZGKcVY8lQ4Zr0b=AW5md-ZGKcVY8lQ4Zr0b
> etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2490) Ensure OzoneClient is closed in Ozone Shell handlers
[
https://issues.apache.org/jira/browse/HDDS-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-2490:
-
Labels: pull-request-available sonar (was: sonar)
> Ensure OzoneClient is closed in Ozone Shell handlers
>
>
> Key: HDDS-2490
> URL: https://issues.apache.org/jira/browse/HDDS-2490
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Reporter: Attila Doroszlai
>Assignee: Prashant Pogde
>Priority: Minor
> Labels: pull-request-available, sonar
>
> OzoneClient should be closed in all command handlers ({{Handler}} subclasses).
> *
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-Y0KcVY8lQ4Zrz6=AW5md-Y0KcVY8lQ4Zrz6
> *
> https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-ZGKcVY8lQ4Zr0b=AW5md-ZGKcVY8lQ4Zr0b
> etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14952) Skip safemode if blockTotal is 0 in new NN
[
https://issues.apache.org/jira/browse/HDFS-14952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976911#comment-16976911
]
Wei-Chiu Chuang commented on HDFS-14952:
+1
> Skip safemode if blockTotal is 0 in new NN
> --
>
> Key: HDFS-14952
> URL: https://issues.apache.org/jira/browse/HDFS-14952
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Rajesh Balamohan
>Assignee: Xiaoqiao He
>Priority: Trivial
> Labels: performance
> Attachments: HDFS-14952.001.patch, HDFS-14952.002.patch,
> HDFS-14952.003.patch
>
>
> When new NN is installed, it spends 30-45 seconds in Safemode. When
> {{blockTotal}} is 0, it should be possible to short circuit safemode check in
> {{BlockManagerSafeMode::areThresholdsMet}}.
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerSafeMode.java#L571
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14940) HDFS Balancer : getBalancerBandwidth displaying wrong values for the maximum network bandwidth used by the datanode while network bandwidth set with values as 104857600
[
https://issues.apache.org/jira/browse/HDFS-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976902#comment-16976902
]
Hadoop QA commented on HDFS-14940:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 0s{color} | {color:orange} hadoop-hdfs-project: The patch generated 4 new +
34 unchanged - 0 fixed = 38 total (was 34) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
39s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 1 new
+ 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
53s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}109m 14s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}197m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
| | org.apache.hadoop.hdfs.protocol.HdfsConstants.MAX_BANDWIDTH_PER_DATANODE
isn't final but should be At HdfsConstants.java:be At
HdfsConstants.java:[line 70] |
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14940 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-14992) TestOfflineEditsViewer is failing in Trunk
[ https://issues.apache.org/jira/browse/HDFS-14992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976866#comment-16976866 ] Surendra Singh Lilhore commented on HDFS-14992: --- Thanks for patch [~hemanthboyina] . Your patch should include binary file. > TestOfflineEditsViewer is failing in Trunk > -- > > Key: HDFS-14992 > URL: https://issues.apache.org/jira/browse/HDFS-14992 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-14992.001.patch, editsStored > > > [https://builds.apache.org/job/PreCommit-HDFS-Build/28310/testReport/] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-2535) TestOzoneManagerDoubleBufferWithOMResponse is flaky
[
https://issues.apache.org/jira/browse/HDDS-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976850#comment-16976850
]
Bharat Viswanadham edited comment on HDDS-2535 at 11/18/19 8:25 PM:
Hi [~elek]
{quote}Independent from the flakiness I think a test where the timeout is 8
minutes and starts 1000 threads to insert 500 buckets (500_000 buckets all
together) it's more like an integration test and would be better to move the
slowest part to the integration-test project.
{quote}
I think now it should run quickly with the fix, and also I think it will not
take that much of time. On my local laptop, I see, it is always completed in
30sec.
And on github run I see it is completed in 53 seconds. I just want to keep this
test in UT, as this will detect any failure in the DoubleBuffer issue which is
a critical component in OM. (Why I want in UT, because we are going to force
sooner, UT should be always green)
[INFO] Running
org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
[1164|https://github.com/bharatviswa504/hadoop-ozone/runs/308637202#step:3:1164][INFO]
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 53.536 s - in
org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
was (Author: bharatviswa):
Hi [~elek]
{quote}Independent from the flakiness I think a test where the timeout is 8
minutes and starts 1000 threads to insert 500 buckets (500_000 buckets all
together) it's more like an integration test and would be better to move the
slowest part to the integration-test project.
{quote}
I think now it should run quickly with the fix, and also I think it will not
take that much of time. On my local laptop, I see, it is always completed in
30sec.
> TestOzoneManagerDoubleBufferWithOMResponse is flaky
> ---
>
> Key: HDDS-2535
> URL: https://issues.apache.org/jira/browse/HDDS-2535
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Marton Elek
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Flakiness can be reproduced locally. Usually it passes, but when I started to
> run it 100 times parallel with high cpu load it failed with the 3rd attempt
> (timed out)
> {code:java}
> ---
> Test set:
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
> ---
> Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 503.297 s <<<
> FAILURE! - in
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
> testDoubleBuffer(org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse)
> Time elapsed: 500.122 s <<< ERROR!
> java.lang.Exception: test timed out after 50 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:382)
> at
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse.testDoubleBuffer(TestOzoneManagerDoubleBufferWithOMResponse.java:385)
> at
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse.testDoubleBuffer(TestOzoneManagerDoubleBufferWithOMResponse.java:129)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
> Independent from the flakiness I think a test where the timeout is 8 minutes
> and starts 1000 threads to insert 500 buckets (500_000 buckets all together)
> it's more like an integration test and would be better to move the slowest
> part to the integration-test project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail:
[jira] [Comment Edited] (HDDS-2535) TestOzoneManagerDoubleBufferWithOMResponse is flaky
[
https://issues.apache.org/jira/browse/HDDS-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976850#comment-16976850
]
Bharat Viswanadham edited comment on HDDS-2535 at 11/18/19 8:25 PM:
Hi [~elek]
{quote}Independent from the flakiness I think a test where the timeout is 8
minutes and starts 1000 threads to insert 500 buckets (500_000 buckets all
together) it's more like an integration test and would be better to move the
slowest part to the integration-test project.
{quote}
I think now it should run quickly with the fix, and also I think it will not
take that much of time. On my local laptop, I see, it is always completed in
30sec.
And on github run I see it is completed in 53 seconds. I just want to keep this
test in UT, as this will detect any failure in the DoubleBuffer issue which is
a critical component in OM. (Why I want in UT, because we are going to force
sooner, UT should be always green)
{code:java}
1164[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
53.536 s - in
org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse{code}
was (Author: bharatviswa):
Hi [~elek]
{quote}Independent from the flakiness I think a test where the timeout is 8
minutes and starts 1000 threads to insert 500 buckets (500_000 buckets all
together) it's more like an integration test and would be better to move the
slowest part to the integration-test project.
{quote}
I think now it should run quickly with the fix, and also I think it will not
take that much of time. On my local laptop, I see, it is always completed in
30sec.
And on github run I see it is completed in 53 seconds. I just want to keep this
test in UT, as this will detect any failure in the DoubleBuffer issue which is
a critical component in OM. (Why I want in UT, because we are going to force
sooner, UT should be always green)
[INFO] Running
org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
[1164|https://github.com/bharatviswa504/hadoop-ozone/runs/308637202#step:3:1164][INFO]
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 53.536 s - in
org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
> TestOzoneManagerDoubleBufferWithOMResponse is flaky
> ---
>
> Key: HDDS-2535
> URL: https://issues.apache.org/jira/browse/HDDS-2535
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Marton Elek
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Flakiness can be reproduced locally. Usually it passes, but when I started to
> run it 100 times parallel with high cpu load it failed with the 3rd attempt
> (timed out)
> {code:java}
> ---
> Test set:
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
> ---
> Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 503.297 s <<<
> FAILURE! - in
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
> testDoubleBuffer(org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse)
> Time elapsed: 500.122 s <<< ERROR!
> java.lang.Exception: test timed out after 50 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:382)
> at
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse.testDoubleBuffer(TestOzoneManagerDoubleBufferWithOMResponse.java:385)
> at
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse.testDoubleBuffer(TestOzoneManagerDoubleBufferWithOMResponse.java:129)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
>
[jira] [Updated] (HDFS-14973) Balancer getBlocks RPC dispersal does not function properly
[
https://issues.apache.org/jira/browse/HDFS-14973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-14973:
---
Attachment: HDFS-14973-branch-2.004.patch
> Balancer getBlocks RPC dispersal does not function properly
> ---
>
> Key: HDFS-14973
> URL: https://issues.apache.org/jira/browse/HDFS-14973
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer mover
>Affects Versions: 2.9.0, 2.7.4, 2.8.2, 3.0.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-14973-branch-2.003.patch,
> HDFS-14973-branch-2.004.patch, HDFS-14973.000.patch, HDFS-14973.001.patch,
> HDFS-14973.002.patch, HDFS-14973.003.patch, HDFS-14973.test.patch
>
>
> In HDFS-11384, a mechanism was added to make the {{getBlocks}} RPC calls
> issued by the Balancer/Mover more dispersed, to alleviate load on the
> NameNode, since {{getBlocks}} can be very expensive and the Balancer should
> not impact normal cluster operation.
> Unfortunately, this functionality does not function as expected, especially
> when the dispatcher thread count is low. The primary issue is that the delay
> is applied only to the first N threads that are submitted to the dispatcher's
> executor, where N is the size of the dispatcher's threadpool, but *not* to
> the first R threads, where R is the number of allowed {{getBlocks}} QPS
> (currently hardcoded to 20). For example, if the threadpool size is 100 (the
> default), threads 0-19 have no delay, 20-99 have increased levels of delay,
> and 100+ have no delay. As I understand it, the intent of the logic was that
> the delay applied to the first 100 threads would force the dispatcher
> executor's threads to all be consumed, thus blocking subsequent (non-delayed)
> threads until the delay period has expired. However, threads 0-19 can finish
> very quickly (their work can often be fulfilled in the time it takes to
> execute a single {{getBlocks}} RPC, on the order of tens of milliseconds),
> thus opening up 20 new slots in the executor, which are then consumed by
> non-delayed threads 100-119, and so on. So, although 80 threads have had a
> delay applied, the non-delay threads rush through in the 20 non-delay slots.
> This problem gets even worse when the dispatcher threadpool size is less than
> the max {{getBlocks}} QPS. For example, if the threadpool size is 10, _no
> threads ever have a delay applied_, and the feature is not enabled at all.
> This problem wasn't surfaced in the original JIRA because the test
> incorrectly measured the period across which {{getBlocks}} RPCs were
> distributed. The variables {{startGetBlocksTime}} and {{endGetBlocksTime}}
> were used to track the time over which the {{getBlocks}} calls were made.
> However, {{startGetBlocksTime}} was initialized at the time of creation of
> the {{FSNameystem}} spy, which is before the mock DataNodes are started. Even
> worse, the Balancer in this test takes 2 iterations to complete balancing the
> cluster, so the time period {{endGetBlocksTime - startGetBlocksTime}}
> actually represents:
> {code}
> (time to submit getBlocks RPCs) + (DataNode startup time) + (time for the
> Dispatcher to complete an iteration of moving blocks)
> {code}
> Thus, the RPC QPS reported by the test is much lower than the RPC QPS seen
> during the period of initial block fetching.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14973) Balancer getBlocks RPC dispersal does not function properly
[
https://issues.apache.org/jira/browse/HDFS-14973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976854#comment-16976854
]
Erik Krogen commented on HDFS-14973:
Attach v004 patch for branch-2 addressing the checkstyle and license issues.
> Balancer getBlocks RPC dispersal does not function properly
> ---
>
> Key: HDFS-14973
> URL: https://issues.apache.org/jira/browse/HDFS-14973
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer mover
>Affects Versions: 2.9.0, 2.7.4, 2.8.2, 3.0.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-14973-branch-2.003.patch,
> HDFS-14973-branch-2.004.patch, HDFS-14973.000.patch, HDFS-14973.001.patch,
> HDFS-14973.002.patch, HDFS-14973.003.patch, HDFS-14973.test.patch
>
>
> In HDFS-11384, a mechanism was added to make the {{getBlocks}} RPC calls
> issued by the Balancer/Mover more dispersed, to alleviate load on the
> NameNode, since {{getBlocks}} can be very expensive and the Balancer should
> not impact normal cluster operation.
> Unfortunately, this functionality does not function as expected, especially
> when the dispatcher thread count is low. The primary issue is that the delay
> is applied only to the first N threads that are submitted to the dispatcher's
> executor, where N is the size of the dispatcher's threadpool, but *not* to
> the first R threads, where R is the number of allowed {{getBlocks}} QPS
> (currently hardcoded to 20). For example, if the threadpool size is 100 (the
> default), threads 0-19 have no delay, 20-99 have increased levels of delay,
> and 100+ have no delay. As I understand it, the intent of the logic was that
> the delay applied to the first 100 threads would force the dispatcher
> executor's threads to all be consumed, thus blocking subsequent (non-delayed)
> threads until the delay period has expired. However, threads 0-19 can finish
> very quickly (their work can often be fulfilled in the time it takes to
> execute a single {{getBlocks}} RPC, on the order of tens of milliseconds),
> thus opening up 20 new slots in the executor, which are then consumed by
> non-delayed threads 100-119, and so on. So, although 80 threads have had a
> delay applied, the non-delay threads rush through in the 20 non-delay slots.
> This problem gets even worse when the dispatcher threadpool size is less than
> the max {{getBlocks}} QPS. For example, if the threadpool size is 10, _no
> threads ever have a delay applied_, and the feature is not enabled at all.
> This problem wasn't surfaced in the original JIRA because the test
> incorrectly measured the period across which {{getBlocks}} RPCs were
> distributed. The variables {{startGetBlocksTime}} and {{endGetBlocksTime}}
> were used to track the time over which the {{getBlocks}} calls were made.
> However, {{startGetBlocksTime}} was initialized at the time of creation of
> the {{FSNameystem}} spy, which is before the mock DataNodes are started. Even
> worse, the Balancer in this test takes 2 iterations to complete balancing the
> cluster, so the time period {{endGetBlocksTime - startGetBlocksTime}}
> actually represents:
> {code}
> (time to submit getBlocks RPCs) + (DataNode startup time) + (time for the
> Dispatcher to complete an iteration of moving blocks)
> {code}
> Thus, the RPC QPS reported by the test is much lower than the RPC QPS seen
> during the period of initial block fetching.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2535) TestOzoneManagerDoubleBufferWithOMResponse is flaky
[
https://issues.apache.org/jira/browse/HDDS-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976850#comment-16976850
]
Bharat Viswanadham commented on HDDS-2535:
--
Hi [~elek]
{quote}Independent from the flakiness I think a test where the timeout is 8
minutes and starts 1000 threads to insert 500 buckets (500_000 buckets all
together) it's more like an integration test and would be better to move the
slowest part to the integration-test project.
{quote}
I think now it should run quickly with the fix, and also I think it will not
take that much of time. On my local laptop, I see, it is always completed in
30sec.
> TestOzoneManagerDoubleBufferWithOMResponse is flaky
> ---
>
> Key: HDDS-2535
> URL: https://issues.apache.org/jira/browse/HDDS-2535
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Marton Elek
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Flakiness can be reproduced locally. Usually it passes, but when I started to
> run it 100 times parallel with high cpu load it failed with the 3rd attempt
> (timed out)
> {code:java}
> ---
> Test set:
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
> ---
> Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 503.297 s <<<
> FAILURE! - in
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
> testDoubleBuffer(org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse)
> Time elapsed: 500.122 s <<< ERROR!
> java.lang.Exception: test timed out after 50 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:382)
> at
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse.testDoubleBuffer(TestOzoneManagerDoubleBufferWithOMResponse.java:385)
> at
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse.testDoubleBuffer(TestOzoneManagerDoubleBufferWithOMResponse.java:129)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
> Independent from the flakiness I think a test where the timeout is 8 minutes
> and starts 1000 threads to insert 500 buckets (500_000 buckets all together)
> it's more like an integration test and would be better to move the slowest
> part to the integration-test project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2535) TestOzoneManagerDoubleBufferWithOMResponse is flaky
[
https://issues.apache.org/jira/browse/HDDS-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-2535:
-
Status: Patch Available (was: Open)
> TestOzoneManagerDoubleBufferWithOMResponse is flaky
> ---
>
> Key: HDDS-2535
> URL: https://issues.apache.org/jira/browse/HDDS-2535
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Marton Elek
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Flakiness can be reproduced locally. Usually it passes, but when I started to
> run it 100 times parallel with high cpu load it failed with the 3rd attempt
> (timed out)
> {code:java}
> ---
> Test set:
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
> ---
> Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 503.297 s <<<
> FAILURE! - in
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
> testDoubleBuffer(org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse)
> Time elapsed: 500.122 s <<< ERROR!
> java.lang.Exception: test timed out after 50 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:382)
> at
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse.testDoubleBuffer(TestOzoneManagerDoubleBufferWithOMResponse.java:385)
> at
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse.testDoubleBuffer(TestOzoneManagerDoubleBufferWithOMResponse.java:129)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
> Independent from the flakiness I think a test where the timeout is 8 minutes
> and starts 1000 threads to insert 500 buckets (500_000 buckets all together)
> it's more like an integration test and would be better to move the slowest
> part to the integration-test project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2535) TestOzoneManagerDoubleBufferWithOMResponse is flaky
[
https://issues.apache.org/jira/browse/HDDS-2535?focusedWorklogId=345552=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345552
]
ASF GitHub Bot logged work on HDDS-2535:
Author: ASF GitHub Bot
Created on: 18/Nov/19 20:01
Start Date: 18/Nov/19 20:01
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #216:
HDDS-2535. TestOzoneManagerDoubleBufferWithOMResponse is flaky.
URL: https://github.com/apache/hadoop-ozone/pull/216
## What changes were proposed in this pull request?
Fix testDoubleBuffer flakiness.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-2535
## How was this patch tested?
Ran the test multiple times, I see now it is passing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 345552)
Remaining Estimate: 0h
Time Spent: 10m
> TestOzoneManagerDoubleBufferWithOMResponse is flaky
> ---
>
> Key: HDDS-2535
> URL: https://issues.apache.org/jira/browse/HDDS-2535
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Marton Elek
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Flakiness can be reproduced locally. Usually it passes, but when I started to
> run it 100 times parallel with high cpu load it failed with the 3rd attempt
> (timed out)
> {code:java}
> ---
> Test set:
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
> ---
> Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 503.297 s <<<
> FAILURE! - in
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
> testDoubleBuffer(org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse)
> Time elapsed: 500.122 s <<< ERROR!
> java.lang.Exception: test timed out after 50 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:382)
> at
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse.testDoubleBuffer(TestOzoneManagerDoubleBufferWithOMResponse.java:385)
> at
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse.testDoubleBuffer(TestOzoneManagerDoubleBufferWithOMResponse.java:129)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
> Independent from the flakiness I think a test where the timeout is 8 minutes
> and starts 1000 threads to insert 500 buckets (500_000 buckets all together)
> it's more like an integration test and would be better to move the slowest
> part to the integration-test project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2459) Refactor ReplicationManager to consider maintenance states
[
https://issues.apache.org/jira/browse/HDDS-2459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976845#comment-16976845
]
Stephen O'Donnell commented on HDDS-2459:
-
With the current "pre decommission" replication logic, the definition of
"isContainerHealthy" is like:
{code}
private boolean isContainerHealthy(final ContainerInfo container,
final Set replicas) {
return container.getReplicationFactor().getNumber() == replicas.size() &&
replicas.stream().allMatch(
r -> compareState(container.getState(), r.getState()));
}
{code}
On one hand, this check will always fail when a container has a maintenance or
decommission replica, as it it will fail the second check as container.getState
!= , but we may want to improve this check to also skip
healthy decom and maintenance containers?
I wonder the check should:
1. Ensure there are no inflight operations
2. The container is sufficiently replicated based on the logic in the comment
above.
3. The container is not over replicated.
4. Any non-maintenance and non-decommission replicas have the same state as the
container itself
Alternatively, we can skip this check and run through the isOverReplicated()
isUnderReplicated and isUnhealthy() scenarios, as we may well need to do about
the same amount of work in the check as we do when checking over / under
replicated or healthy later anyway.
> Refactor ReplicationManager to consider maintenance states
> --
>
> Key: HDDS-2459
> URL: https://issues.apache.org/jira/browse/HDDS-2459
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: SCM
>Affects Versions: 0.5.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
>
> In its current form the replication manager does not consider decommission or
> maintenance states when checking if replicas are sufficiently replicated.
> With the introduction of maintenance states, it needs to consider
> decommission and maintenance states when deciding if blocks are over or under
> replicated.
> It also needs to provide an API to allow the decommission manager to check if
> blocks are over or under replicated, so the decommission manager can decide
> if a node has completed decommission and maintenance or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2535) TestOzoneManagerDoubleBufferWithOMResponse is flaky
[
https://issues.apache.org/jira/browse/HDDS-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-2535:
-
Labels: pull-request-available (was: )
> TestOzoneManagerDoubleBufferWithOMResponse is flaky
> ---
>
> Key: HDDS-2535
> URL: https://issues.apache.org/jira/browse/HDDS-2535
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Marton Elek
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
>
> Flakiness can be reproduced locally. Usually it passes, but when I started to
> run it 100 times parallel with high cpu load it failed with the 3rd attempt
> (timed out)
> {code:java}
> ---
> Test set:
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
> ---
> Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 503.297 s <<<
> FAILURE! - in
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
> testDoubleBuffer(org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse)
> Time elapsed: 500.122 s <<< ERROR!
> java.lang.Exception: test timed out after 50 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:382)
> at
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse.testDoubleBuffer(TestOzoneManagerDoubleBufferWithOMResponse.java:385)
> at
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse.testDoubleBuffer(TestOzoneManagerDoubleBufferWithOMResponse.java:129)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
> Independent from the flakiness I think a test where the timeout is 8 minutes
> and starts 1000 threads to insert 500 buckets (500_000 buckets all together)
> it's more like an integration test and would be better to move the slowest
> part to the integration-test project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2535) TestOzoneManagerDoubleBufferWithOMResponse is flaky
[
https://issues.apache.org/jira/browse/HDDS-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham reassigned HDDS-2535:
Assignee: Bharat Viswanadham
> TestOzoneManagerDoubleBufferWithOMResponse is flaky
> ---
>
> Key: HDDS-2535
> URL: https://issues.apache.org/jira/browse/HDDS-2535
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Marton Elek
>Assignee: Bharat Viswanadham
>Priority: Major
>
> Flakiness can be reproduced locally. Usually it passes, but when I started to
> run it 100 times parallel with high cpu load it failed with the 3rd attempt
> (timed out)
> {code:java}
> ---
> Test set:
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
> ---
> Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 503.297 s <<<
> FAILURE! - in
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
> testDoubleBuffer(org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse)
> Time elapsed: 500.122 s <<< ERROR!
> java.lang.Exception: test timed out after 50 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:382)
> at
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse.testDoubleBuffer(TestOzoneManagerDoubleBufferWithOMResponse.java:385)
> at
> org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse.testDoubleBuffer(TestOzoneManagerDoubleBufferWithOMResponse.java:129)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
> Independent from the flakiness I think a test where the timeout is 8 minutes
> and starts 1000 threads to insert 500 buckets (500_000 buckets all together)
> it's more like an integration test and would be better to move the slowest
> part to the integration-test project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14973) Balancer getBlocks RPC dispersal does not function properly
[
https://issues.apache.org/jira/browse/HDFS-14973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976826#comment-16976826
]
Hadoop QA commented on HDFS-14973:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
4s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 644 unchanged - 1 fixed = 646 total (was 645) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 16s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
28s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}110m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:f555aa740b5 |
| JIRA Issue | HDFS-14973 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12986141/HDFS-14973-branch-2.003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 3502fbd10de3 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019
[jira] [Commented] (HDFS-14969) Fix HDFS client unnecessary failover log printing
[
https://issues.apache.org/jira/browse/HDFS-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976810#comment-16976810
]
Wei-Chiu Chuang commented on HDFS-14969:
i would love to see the client being more intelligent about when to emit a
message like this. Thanks for doing this.
> Fix HDFS client unnecessary failover log printing
> -
>
> Key: HDFS-14969
> URL: https://issues.apache.org/jira/browse/HDFS-14969
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 3.1.3
>Reporter: Xudong Cao
>Assignee: Xudong Cao
>Priority: Minor
> Labels: multi-sbnn
>
> In multi-NameNodes scenario, suppose there are 3 NNs and the 3rd is ANN, and
> then a client starts rpc with the 1st NN, it will be silent when failover
> from the 1st NN to the 2nd NN, but when failover from the 2nd NN to the 3rd
> NN, it prints some unnecessary logs, in some scenarios, these logs will be
> very numerous:
> {code:java}
> 2019-11-07 11:35:41,577 INFO retry.RetryInvocationHandler:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> Operation category READ is not supported in state standby. Visit
> https://s.apache.org/sbnn-error
> at
> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:98)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2052)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1459)
> ...{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14940) HDFS Balancer : getBalancerBandwidth displaying wrong values for the maximum network bandwidth used by the datanode while network bandwidth set with values as 104857600
[ https://issues.apache.org/jira/browse/HDFS-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976806#comment-16976806 ] Surendra Singh Lilhore commented on HDFS-14940: --- Thanks [~hemanthboyina] for patch. minor comment. # Throw *IllegalArgumentException*. # Allow 1TB in bandwidth. So use only ">" for condition check. > HDFS Balancer : getBalancerBandwidth displaying wrong values for the maximum > network bandwidth used by the datanode while network bandwidth set with > values as 1048576000g/1048p/1e > --- > > Key: HDFS-14940 > URL: https://issues.apache.org/jira/browse/HDFS-14940 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer mover >Affects Versions: 3.1.1 > Environment: 3 Node HA Setup >Reporter: Souryakanta Dwivedy >Priority: Minor > Attachments: BalancerBW.PNG, HDFS-14940.001.patch, > HDFS-14940.002.patch > > > HDFS Balancer : getBalancerBandwidth displaying wrong values for the maximum > network bandwidth used by the datanode > while network bandwidth set with values as 1048576000g/1048p/1e > Steps :- > * Set balancer bandwith with command setBalancerBandwidth and vlaues as > [1048576000g/1048p/1e] > * - Check bandwidth used by the datanode during HDFS block balancing with > command :hdfs dfsadmin -getBalancerBandwidth " check it will display some > different values not the same value as set -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14969) Fix HDFS client unnecessary failover log printing
[
https://issues.apache.org/jira/browse/HDFS-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14969:
---
Labels: multi-sbnn (was: )
> Fix HDFS client unnecessary failover log printing
> -
>
> Key: HDFS-14969
> URL: https://issues.apache.org/jira/browse/HDFS-14969
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 3.1.3
>Reporter: Xudong Cao
>Assignee: Xudong Cao
>Priority: Minor
> Labels: multi-sbnn
>
> In multi-NameNodes scenario, suppose there are 3 NNs and the 3rd is ANN, and
> then a client starts rpc with the 1st NN, it will be silent when failover
> from the 1st NN to the 2nd NN, but when failover from the 2nd NN to the 3rd
> NN, it prints some unnecessary logs, in some scenarios, these logs will be
> very numerous:
> {code:java}
> 2019-11-07 11:35:41,577 INFO retry.RetryInvocationHandler:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> Operation category READ is not supported in state standby. Visit
> https://s.apache.org/sbnn-error
> at
> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:98)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2052)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1459)
> ...{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2523) BufferPool.releaseBuffer may release a buffer different than the head of the list
[
https://issues.apache.org/jira/browse/HDDS-2523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976771#comment-16976771
]
Tsz-wo Sze commented on HDDS-2523:
--
a.patch: for debug
It expected ChunkBufferImplWithByteBuffer3 but removed
ChunkBufferImplWithByteBuffer1.
{code}
Before remove(0): [ChunkBufferImplWithByteBuffer1:limit=1048576,
ChunkBufferImplWithByteBuffer3:limit=1048576]
chunkBuffer: ChunkBufferImplWithByteBuffer3:limit=1048576
buffer : ChunkBufferImplWithByteBuffer1:limit=1048576
{code}
> BufferPool.releaseBuffer may release a buffer different than the head of the
> list
> -
>
> Key: HDDS-2523
> URL: https://issues.apache.org/jira/browse/HDDS-2523
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Reporter: Tsz-wo Sze
>Assignee: Attila Doroszlai
>Priority: Major
> Attachments: a.patch
>
>
> {code}
> //BufferPool
> public void releaseBuffer(ByteBuffer byteBuffer) {
> // always remove from head of the list and append at last
> ByteBuffer buffer = bufferList.remove(0);
> // Ensure the buffer to be removed is always at the head of the list.
> Preconditions.checkArgument(buffer.equals(byteBuffer));
> buffer.clear();
> bufferList.add(buffer);
> Preconditions.checkArgument(currentBufferIndex >= 0);
> currentBufferIndex--;
> }
> {code}
> In the code above, it expects buffer and byteBuffer are the same object, i.e.
> buffer == byteBuffer. However the precondition is checking
> buffer.equals(byteBuffer). Unfortunately, both buffer and byteBuffer have
> remaining() == 0 so that equals(..) returns true and the precondition does
> not catch the bug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2523) BufferPool.releaseBuffer may release a buffer different than the head of the list
[
https://issues.apache.org/jira/browse/HDDS-2523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz-wo Sze updated HDDS-2523:
-
Attachment: a.patch
> BufferPool.releaseBuffer may release a buffer different than the head of the
> list
> -
>
> Key: HDDS-2523
> URL: https://issues.apache.org/jira/browse/HDDS-2523
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Reporter: Tsz-wo Sze
>Assignee: Attila Doroszlai
>Priority: Major
> Attachments: a.patch
>
>
> {code}
> //BufferPool
> public void releaseBuffer(ByteBuffer byteBuffer) {
> // always remove from head of the list and append at last
> ByteBuffer buffer = bufferList.remove(0);
> // Ensure the buffer to be removed is always at the head of the list.
> Preconditions.checkArgument(buffer.equals(byteBuffer));
> buffer.clear();
> bufferList.add(buffer);
> Preconditions.checkArgument(currentBufferIndex >= 0);
> currentBufferIndex--;
> }
> {code}
> In the code above, it expects buffer and byteBuffer are the same object, i.e.
> buffer == byteBuffer. However the precondition is checking
> buffer.equals(byteBuffer). Unfortunately, both buffer and byteBuffer have
> remaining() == 0 so that equals(..) returns true and the precondition does
> not catch the bug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14942) Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndex
[
https://issues.apache.org/jira/browse/HDFS-14942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14942:
---
Fix Version/s: 3.2.2
3.1.4
> Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndex
> -
>
> Key: HDFS-14942
> URL: https://issues.apache.org/jira/browse/HDFS-14942
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Minor
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-14942.001.patch, HDFS-14942.002.patch,
> HDFS-14942.003.patch, HDFS-14942.004.patch
>
>
> when hadoop 2.x upgrades to hadoop 3.x, InterQJournalProtocol is newly
> added，so throw Unknown protocol.
> the newly InterQJournalProtocol is used to sychronize past log segments to
> JNs that missed them. And that an error occurs does not affect normal
> service. I think it should not be a ERROR log，and that log a warn log is more
> reasonable.
> {code:java}
> private void syncWithJournalAtIndex(int index) {
> ...
> GetEditLogManifestResponseProto editLogManifest;
> try {
> editLogManifest = jnProxy.getEditLogManifestFromJournal(jid,
> nameServiceId, 0, false);
> } catch (IOException e) {
> LOG.error("Could not sync with Journal at " +
> otherJNProxies.get(journalNodeIndexForSync), e);
> return;
> }
> {code}
> {code:java}
> 2019-10-30,15:11:17,388 ERROR
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer: Could not sync with
> Journal at mos1-hadoop-prc-ct17.ksru/10.85.3.59:111002019-10-30,15:11:17,388
> ERROR org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer: Could not
> sync with Journal at
> mos1-hadoop-prc-ct17.ksru/10.85.3.59:11100org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> Unknown protocol:
> org.apache.hadoop.hdfs.qjournal.protocol.InterQJournalProtocol
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1565)
> at org.apache.hadoop.ipc.Client.call(Client.java:1511)
> at org.apache.hadoop.ipc.Client.call(Client.java:1421)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy16.getEditLogManifestFromJournal(Unknown Source)
> at
> org.apache.hadoop.hdfs.qjournal.protocolPB.InterQJournalProtocolTranslatorPB.getEditLogManifestFromJournal(InterQJournalProtocolTranslatorPB.java:75)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncWithJournalAtIndex(JournalNodeSyncer.java:250)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncJournals(JournalNodeSyncer.java:226)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.lambda$startSyncJournalsDaemon$0(JournalNodeSyncer.java:186)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-14942) Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndex
[
https://issues.apache.org/jira/browse/HDFS-14942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14942:
---
Comment: was deleted
(was: I think we want this in 2.x not 3.x)
> Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndex
> -
>
> Key: HDFS-14942
> URL: https://issues.apache.org/jira/browse/HDFS-14942
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HDFS-14942.001.patch, HDFS-14942.002.patch,
> HDFS-14942.003.patch, HDFS-14942.004.patch
>
>
> when hadoop 2.x upgrades to hadoop 3.x, InterQJournalProtocol is newly
> added，so throw Unknown protocol.
> the newly InterQJournalProtocol is used to sychronize past log segments to
> JNs that missed them. And that an error occurs does not affect normal
> service. I think it should not be a ERROR log，and that log a warn log is more
> reasonable.
> {code:java}
> private void syncWithJournalAtIndex(int index) {
> ...
> GetEditLogManifestResponseProto editLogManifest;
> try {
> editLogManifest = jnProxy.getEditLogManifestFromJournal(jid,
> nameServiceId, 0, false);
> } catch (IOException e) {
> LOG.error("Could not sync with Journal at " +
> otherJNProxies.get(journalNodeIndexForSync), e);
> return;
> }
> {code}
> {code:java}
> 2019-10-30,15:11:17,388 ERROR
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer: Could not sync with
> Journal at mos1-hadoop-prc-ct17.ksru/10.85.3.59:111002019-10-30,15:11:17,388
> ERROR org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer: Could not
> sync with Journal at
> mos1-hadoop-prc-ct17.ksru/10.85.3.59:11100org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> Unknown protocol:
> org.apache.hadoop.hdfs.qjournal.protocol.InterQJournalProtocol
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1565)
> at org.apache.hadoop.ipc.Client.call(Client.java:1511)
> at org.apache.hadoop.ipc.Client.call(Client.java:1421)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy16.getEditLogManifestFromJournal(Unknown Source)
> at
> org.apache.hadoop.hdfs.qjournal.protocolPB.InterQJournalProtocolTranslatorPB.getEditLogManifestFromJournal(InterQJournalProtocolTranslatorPB.java:75)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncWithJournalAtIndex(JournalNodeSyncer.java:250)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncJournals(JournalNodeSyncer.java:226)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.lambda$startSyncJournalsDaemon$0(JournalNodeSyncer.java:186)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14942) Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndex
[
https://issues.apache.org/jira/browse/HDFS-14942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976766#comment-16976766
]
Wei-Chiu Chuang commented on HDFS-14942:
I think we want this in 2.x not 3.x
> Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndex
> -
>
> Key: HDFS-14942
> URL: https://issues.apache.org/jira/browse/HDFS-14942
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HDFS-14942.001.patch, HDFS-14942.002.patch,
> HDFS-14942.003.patch, HDFS-14942.004.patch
>
>
> when hadoop 2.x upgrades to hadoop 3.x, InterQJournalProtocol is newly
> added，so throw Unknown protocol.
> the newly InterQJournalProtocol is used to sychronize past log segments to
> JNs that missed them. And that an error occurs does not affect normal
> service. I think it should not be a ERROR log，and that log a warn log is more
> reasonable.
> {code:java}
> private void syncWithJournalAtIndex(int index) {
> ...
> GetEditLogManifestResponseProto editLogManifest;
> try {
> editLogManifest = jnProxy.getEditLogManifestFromJournal(jid,
> nameServiceId, 0, false);
> } catch (IOException e) {
> LOG.error("Could not sync with Journal at " +
> otherJNProxies.get(journalNodeIndexForSync), e);
> return;
> }
> {code}
> {code:java}
> 2019-10-30,15:11:17,388 ERROR
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer: Could not sync with
> Journal at mos1-hadoop-prc-ct17.ksru/10.85.3.59:111002019-10-30,15:11:17,388
> ERROR org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer: Could not
> sync with Journal at
> mos1-hadoop-prc-ct17.ksru/10.85.3.59:11100org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> Unknown protocol:
> org.apache.hadoop.hdfs.qjournal.protocol.InterQJournalProtocol
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1565)
> at org.apache.hadoop.ipc.Client.call(Client.java:1511)
> at org.apache.hadoop.ipc.Client.call(Client.java:1421)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy16.getEditLogManifestFromJournal(Unknown Source)
> at
> org.apache.hadoop.hdfs.qjournal.protocolPB.InterQJournalProtocolTranslatorPB.getEditLogManifestFromJournal(InterQJournalProtocolTranslatorPB.java:75)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncWithJournalAtIndex(JournalNodeSyncer.java:250)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncJournals(JournalNodeSyncer.java:226)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.lambda$startSyncJournalsDaemon$0(JournalNodeSyncer.java:186)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2523) BufferPool.releaseBuffer may release a buffer different than the head of the list
[
https://issues.apache.org/jira/browse/HDDS-2523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976761#comment-16976761
]
Attila Doroszlai commented on HDDS-2523:
Thanks [~szetszwo], that would be great. I also debugged it and found so far
that this happens with standalone replication type (which TestContainerMapper
is using). I have a simple fix idea, but need to test it with other cases, too.
> BufferPool.releaseBuffer may release a buffer different than the head of the
> list
> -
>
> Key: HDDS-2523
> URL: https://issues.apache.org/jira/browse/HDDS-2523
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Reporter: Tsz-wo Sze
>Assignee: Attila Doroszlai
>Priority: Major
>
> {code}
> //BufferPool
> public void releaseBuffer(ByteBuffer byteBuffer) {
> // always remove from head of the list and append at last
> ByteBuffer buffer = bufferList.remove(0);
> // Ensure the buffer to be removed is always at the head of the list.
> Preconditions.checkArgument(buffer.equals(byteBuffer));
> buffer.clear();
> bufferList.add(buffer);
> Preconditions.checkArgument(currentBufferIndex >= 0);
> currentBufferIndex--;
> }
> {code}
> In the code above, it expects buffer and byteBuffer are the same object, i.e.
> buffer == byteBuffer. However the precondition is checking
> buffer.equals(byteBuffer). Unfortunately, both buffer and byteBuffer have
> remaining() == 0 so that equals(..) returns true and the precondition does
> not catch the bug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14940) HDFS Balancer : getBalancerBandwidth displaying wrong values for the maximum network bandwidth used by the datanode while network bandwidth set with values as 1048576000g
[ https://issues.apache.org/jira/browse/HDFS-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina updated HDFS-14940: - Attachment: HDFS-14940.002.patch > HDFS Balancer : getBalancerBandwidth displaying wrong values for the maximum > network bandwidth used by the datanode while network bandwidth set with > values as 1048576000g/1048p/1e > --- > > Key: HDFS-14940 > URL: https://issues.apache.org/jira/browse/HDFS-14940 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer mover >Affects Versions: 3.1.1 > Environment: 3 Node HA Setup >Reporter: Souryakanta Dwivedy >Priority: Minor > Attachments: BalancerBW.PNG, HDFS-14940.001.patch, > HDFS-14940.002.patch > > > HDFS Balancer : getBalancerBandwidth displaying wrong values for the maximum > network bandwidth used by the datanode > while network bandwidth set with values as 1048576000g/1048p/1e > Steps :- > * Set balancer bandwith with command setBalancerBandwidth and vlaues as > [1048576000g/1048p/1e] > * - Check bandwidth used by the datanode during HDFS block balancing with > command :hdfs dfsadmin -getBalancerBandwidth " check it will display some > different values not the same value as set -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2523) BufferPool.releaseBuffer may release a buffer different than the head of the list
[
https://issues.apache.org/jira/browse/HDDS-2523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976756#comment-16976756
]
Tsz-wo Sze commented on HDDS-2523:
--
[~adoroszlai], thanks for working on this. I could post some debug I already
have.
> BufferPool.releaseBuffer may release a buffer different than the head of the
> list
> -
>
> Key: HDDS-2523
> URL: https://issues.apache.org/jira/browse/HDDS-2523
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Tsz-wo Sze
>Assignee: Attila Doroszlai
>Priority: Major
>
> {code}
> //BufferPool
> public void releaseBuffer(ByteBuffer byteBuffer) {
> // always remove from head of the list and append at last
> ByteBuffer buffer = bufferList.remove(0);
> // Ensure the buffer to be removed is always at the head of the list.
> Preconditions.checkArgument(buffer.equals(byteBuffer));
> buffer.clear();
> bufferList.add(buffer);
> Preconditions.checkArgument(currentBufferIndex >= 0);
> currentBufferIndex--;
> }
> {code}
> In the code above, it expects buffer and byteBuffer are the same object, i.e.
> buffer == byteBuffer. However the precondition is checking
> buffer.equals(byteBuffer). Unfortunately, both buffer and byteBuffer have
> remaining() == 0 so that equals(..) returns true and the precondition does
> not catch the bug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2523) BufferPool.releaseBuffer may release a buffer different than the head of the list
[
https://issues.apache.org/jira/browse/HDDS-2523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz-wo Sze updated HDDS-2523:
-
Component/s: Ozone Client
> BufferPool.releaseBuffer may release a buffer different than the head of the
> list
> -
>
> Key: HDDS-2523
> URL: https://issues.apache.org/jira/browse/HDDS-2523
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Reporter: Tsz-wo Sze
>Assignee: Attila Doroszlai
>Priority: Major
>
> {code}
> //BufferPool
> public void releaseBuffer(ByteBuffer byteBuffer) {
> // always remove from head of the list and append at last
> ByteBuffer buffer = bufferList.remove(0);
> // Ensure the buffer to be removed is always at the head of the list.
> Preconditions.checkArgument(buffer.equals(byteBuffer));
> buffer.clear();
> bufferList.add(buffer);
> Preconditions.checkArgument(currentBufferIndex >= 0);
> currentBufferIndex--;
> }
> {code}
> In the code above, it expects buffer and byteBuffer are the same object, i.e.
> buffer == byteBuffer. However the precondition is checking
> buffer.equals(byteBuffer). Unfortunately, both buffer and byteBuffer have
> remaining() == 0 so that equals(..) returns true and the precondition does
> not catch the bug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDDS-2523) BufferPool.releaseBuffer may release a buffer different than the head of the list
[
https://issues.apache.org/jira/browse/HDDS-2523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDDS-2523 started by Attila Doroszlai.
--
> BufferPool.releaseBuffer may release a buffer different than the head of the
> list
> -
>
> Key: HDDS-2523
> URL: https://issues.apache.org/jira/browse/HDDS-2523
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Tsz-wo Sze
>Assignee: Attila Doroszlai
>Priority: Major
>
> {code}
> //BufferPool
> public void releaseBuffer(ByteBuffer byteBuffer) {
> // always remove from head of the list and append at last
> ByteBuffer buffer = bufferList.remove(0);
> // Ensure the buffer to be removed is always at the head of the list.
> Preconditions.checkArgument(buffer.equals(byteBuffer));
> buffer.clear();
> bufferList.add(buffer);
> Preconditions.checkArgument(currentBufferIndex >= 0);
> currentBufferIndex--;
> }
> {code}
> In the code above, it expects buffer and byteBuffer are the same object, i.e.
> buffer == byteBuffer. However the precondition is checking
> buffer.equals(byteBuffer). Unfortunately, both buffer and byteBuffer have
> remaining() == 0 so that equals(..) returns true and the precondition does
> not catch the bug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2523) BufferPool.releaseBuffer may release a buffer different than the head of the list
[
https://issues.apache.org/jira/browse/HDDS-2523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai reassigned HDDS-2523:
--
Assignee: Attila Doroszlai
> BufferPool.releaseBuffer may release a buffer different than the head of the
> list
> -
>
> Key: HDDS-2523
> URL: https://issues.apache.org/jira/browse/HDDS-2523
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Tsz-wo Sze
>Assignee: Attila Doroszlai
>Priority: Major
>
> {code}
> //BufferPool
> public void releaseBuffer(ByteBuffer byteBuffer) {
> // always remove from head of the list and append at last
> ByteBuffer buffer = bufferList.remove(0);
> // Ensure the buffer to be removed is always at the head of the list.
> Preconditions.checkArgument(buffer.equals(byteBuffer));
> buffer.clear();
> bufferList.add(buffer);
> Preconditions.checkArgument(currentBufferIndex >= 0);
> currentBufferIndex--;
> }
> {code}
> In the code above, it expects buffer and byteBuffer are the same object, i.e.
> buffer == byteBuffer. However the precondition is checking
> buffer.equals(byteBuffer). Unfortunately, both buffer and byteBuffer have
> remaining() == 0 so that equals(..) returns true and the precondition does
> not catch the bug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14992) TestOfflineEditsViewer is failing in Trunk
[
https://issues.apache.org/jira/browse/HDFS-14992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976734#comment-16976734
]
Hadoop QA commented on HDFS-14992:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
32m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 30s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
56s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14992 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12986138/HDFS-14992.001.patch |
| Optional Tests | dupname asflicense unit xml |
| uname | Linux 9d2377f1faac 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2764236 |
| maven | version: Apache Maven 3.3.9 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28326/testReport/ |
| Max. process+thread count | 310 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/28326/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> TestOfflineEditsViewer is failing in Trunk
> --
>
> Key: HDFS-14992
> URL: https://issues.apache.org/jira/browse/HDFS-14992
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: hemanthboyina
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HDFS-14992.001.patch, editsStored
>
>
> [https://builds.apache.org/job/PreCommit-HDFS-Build/28310/testReport/]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14974) RBF: Make tests use free ports
[
https://issues.apache.org/jira/browse/HDFS-14974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976732#comment-16976732
]
Íñigo Goiri commented on HDFS-14974:
Thanks [~ayushtkn] for the commit!
> RBF: Make tests use free ports
> --
>
> Key: HDFS-14974
> URL: https://issues.apache.org/jira/browse/HDFS-14974
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14974.000.patch
>
>
> Currently, {{TestRouterSecurityManager#testCreateCredentials}} create a
> Router with the default ports. However, these ports might be used. We should
> set it to :0 for it to be assigned dynamically.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14992) TestOfflineEditsViewer is failing in Trunk
[ https://issues.apache.org/jira/browse/HDFS-14992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976731#comment-16976731 ] Íñigo Goiri commented on HDFS-14992: [~hemanthboyina], thanks for fixing this. Can you link the JIRA that triggered this? > TestOfflineEditsViewer is failing in Trunk > -- > > Key: HDFS-14992 > URL: https://issues.apache.org/jira/browse/HDFS-14992 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-14992.001.patch, editsStored > > > [https://builds.apache.org/job/PreCommit-HDFS-Build/28310/testReport/] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14973) Balancer getBlocks RPC dispersal does not function properly
[
https://issues.apache.org/jira/browse/HDFS-14973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976729#comment-16976729
]
Erik Krogen commented on HDFS-14973:
Just attached a branch-2 patch. There are two changes from the 3.x line:
* Removed the use of {{AtomicLong#getAndUpdate}} within {{TestBalancer}} in
favor of the more primitive {{compareAndSet}} to be Java 7 compatible.
* Remove the use of Guava's {{RateLimiter}}, not present in the version of
Guava used in branch-2. Implemented a simple {{RateLimiter}} that disperses
calls by ensuring they are separated by least (1/rate) time. This can slightly
over-limit. For example with a 10 QPS rate limit, in the following scenario,
thread B will be forced to wait 100ms. I don't think it's an issue in this
case, as all of the dispatch threads will be issued simultaneously, so after a
single initial pause they will all attempt to submit their RPC and be nicely
smoothed out.
** 500ms passes with no RPCs
** thread A submits an RPC
** thread B immediately tries to submit an RPC
[~shv], let me know what you think.
> Balancer getBlocks RPC dispersal does not function properly
> ---
>
> Key: HDFS-14973
> URL: https://issues.apache.org/jira/browse/HDFS-14973
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer mover
>Affects Versions: 2.9.0, 2.7.4, 2.8.2, 3.0.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-14973-branch-2.003.patch, HDFS-14973.000.patch,
> HDFS-14973.001.patch, HDFS-14973.002.patch, HDFS-14973.003.patch,
> HDFS-14973.test.patch
>
>
> In HDFS-11384, a mechanism was added to make the {{getBlocks}} RPC calls
> issued by the Balancer/Mover more dispersed, to alleviate load on the
> NameNode, since {{getBlocks}} can be very expensive and the Balancer should
> not impact normal cluster operation.
> Unfortunately, this functionality does not function as expected, especially
> when the dispatcher thread count is low. The primary issue is that the delay
> is applied only to the first N threads that are submitted to the dispatcher's
> executor, where N is the size of the dispatcher's threadpool, but *not* to
> the first R threads, where R is the number of allowed {{getBlocks}} QPS
> (currently hardcoded to 20). For example, if the threadpool size is 100 (the
> default), threads 0-19 have no delay, 20-99 have increased levels of delay,
> and 100+ have no delay. As I understand it, the intent of the logic was that
> the delay applied to the first 100 threads would force the dispatcher
> executor's threads to all be consumed, thus blocking subsequent (non-delayed)
> threads until the delay period has expired. However, threads 0-19 can finish
> very quickly (their work can often be fulfilled in the time it takes to
> execute a single {{getBlocks}} RPC, on the order of tens of milliseconds),
> thus opening up 20 new slots in the executor, which are then consumed by
> non-delayed threads 100-119, and so on. So, although 80 threads have had a
> delay applied, the non-delay threads rush through in the 20 non-delay slots.
> This problem gets even worse when the dispatcher threadpool size is less than
> the max {{getBlocks}} QPS. For example, if the threadpool size is 10, _no
> threads ever have a delay applied_, and the feature is not enabled at all.
> This problem wasn't surfaced in the original JIRA because the test
> incorrectly measured the period across which {{getBlocks}} RPCs were
> distributed. The variables {{startGetBlocksTime}} and {{endGetBlocksTime}}
> were used to track the time over which the {{getBlocks}} calls were made.
> However, {{startGetBlocksTime}} was initialized at the time of creation of
> the {{FSNameystem}} spy, which is before the mock DataNodes are started. Even
> worse, the Balancer in this test takes 2 iterations to complete balancing the
> cluster, so the time period {{endGetBlocksTime - startGetBlocksTime}}
> actually represents:
> {code}
> (time to submit getBlocks RPCs) + (DataNode startup time) + (time for the
> Dispatcher to complete an iteration of moving blocks)
> {code}
> Thus, the RPC QPS reported by the test is much lower than the RPC QPS seen
> during the period of initial block fetching.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14992) TestOfflineEditsViewer is failing in Trunk
[ https://issues.apache.org/jira/browse/HDFS-14992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina updated HDFS-14992: - Status: Open (was: Patch Available) > TestOfflineEditsViewer is failing in Trunk > -- > > Key: HDFS-14992 > URL: https://issues.apache.org/jira/browse/HDFS-14992 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-14992.001.patch, editsStored > > > [https://builds.apache.org/job/PreCommit-HDFS-Build/28310/testReport/] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14992) TestOfflineEditsViewer is failing in Trunk
[ https://issues.apache.org/jira/browse/HDFS-14992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina updated HDFS-14992: - Status: Patch Available (was: Open) > TestOfflineEditsViewer is failing in Trunk > -- > > Key: HDFS-14992 > URL: https://issues.apache.org/jira/browse/HDFS-14992 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-14992.001.patch, editsStored > > > [https://builds.apache.org/job/PreCommit-HDFS-Build/28310/testReport/] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14973) Balancer getBlocks RPC dispersal does not function properly
[
https://issues.apache.org/jira/browse/HDFS-14973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-14973:
---
Attachment: HDFS-14973-branch-2.003.patch
> Balancer getBlocks RPC dispersal does not function properly
> ---
>
> Key: HDFS-14973
> URL: https://issues.apache.org/jira/browse/HDFS-14973
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer mover
>Affects Versions: 2.9.0, 2.7.4, 2.8.2, 3.0.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-14973-branch-2.003.patch, HDFS-14973.000.patch,
> HDFS-14973.001.patch, HDFS-14973.002.patch, HDFS-14973.003.patch,
> HDFS-14973.test.patch
>
>
> In HDFS-11384, a mechanism was added to make the {{getBlocks}} RPC calls
> issued by the Balancer/Mover more dispersed, to alleviate load on the
> NameNode, since {{getBlocks}} can be very expensive and the Balancer should
> not impact normal cluster operation.
> Unfortunately, this functionality does not function as expected, especially
> when the dispatcher thread count is low. The primary issue is that the delay
> is applied only to the first N threads that are submitted to the dispatcher's
> executor, where N is the size of the dispatcher's threadpool, but *not* to
> the first R threads, where R is the number of allowed {{getBlocks}} QPS
> (currently hardcoded to 20). For example, if the threadpool size is 100 (the
> default), threads 0-19 have no delay, 20-99 have increased levels of delay,
> and 100+ have no delay. As I understand it, the intent of the logic was that
> the delay applied to the first 100 threads would force the dispatcher
> executor's threads to all be consumed, thus blocking subsequent (non-delayed)
> threads until the delay period has expired. However, threads 0-19 can finish
> very quickly (their work can often be fulfilled in the time it takes to
> execute a single {{getBlocks}} RPC, on the order of tens of milliseconds),
> thus opening up 20 new slots in the executor, which are then consumed by
> non-delayed threads 100-119, and so on. So, although 80 threads have had a
> delay applied, the non-delay threads rush through in the 20 non-delay slots.
> This problem gets even worse when the dispatcher threadpool size is less than
> the max {{getBlocks}} QPS. For example, if the threadpool size is 10, _no
> threads ever have a delay applied_, and the feature is not enabled at all.
> This problem wasn't surfaced in the original JIRA because the test
> incorrectly measured the period across which {{getBlocks}} RPCs were
> distributed. The variables {{startGetBlocksTime}} and {{endGetBlocksTime}}
> were used to track the time over which the {{getBlocks}} calls were made.
> However, {{startGetBlocksTime}} was initialized at the time of creation of
> the {{FSNameystem}} spy, which is before the mock DataNodes are started. Even
> worse, the Balancer in this test takes 2 iterations to complete balancing the
> cluster, so the time period {{endGetBlocksTime - startGetBlocksTime}}
> actually represents:
> {code}
> (time to submit getBlocks RPCs) + (DataNode startup time) + (time for the
> Dispatcher to complete an iteration of moving blocks)
> {code}
> Thus, the RPC QPS reported by the test is much lower than the RPC QPS seen
> during the period of initial block fetching.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14992) TestOfflineEditsViewer is failing in Trunk
[
https://issues.apache.org/jira/browse/HDFS-14992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976709#comment-16976709
]
Hadoop QA commented on HDFS-14992:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HDFS-14992 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14992 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/28327/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> TestOfflineEditsViewer is failing in Trunk
> --
>
> Key: HDFS-14992
> URL: https://issues.apache.org/jira/browse/HDFS-14992
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: hemanthboyina
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HDFS-14992.001.patch, editsStored
>
>
> [https://builds.apache.org/job/PreCommit-HDFS-Build/28310/testReport/]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14992) TestOfflineEditsViewer is failing in Trunk
[ https://issues.apache.org/jira/browse/HDFS-14992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina updated HDFS-14992: - Attachment: editsStored > TestOfflineEditsViewer is failing in Trunk > -- > > Key: HDFS-14992 > URL: https://issues.apache.org/jira/browse/HDFS-14992 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-14992.001.patch, editsStored > > > [https://builds.apache.org/job/PreCommit-HDFS-Build/28310/testReport/] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14992) TestOfflineEditsViewer is failing in Trunk
[ https://issues.apache.org/jira/browse/HDFS-14992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina updated HDFS-14992: - Attachment: HDFS-14992.001.patch Status: Patch Available (was: Open) > TestOfflineEditsViewer is failing in Trunk > -- > > Key: HDFS-14992 > URL: https://issues.apache.org/jira/browse/HDFS-14992 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-14992.001.patch > > > [https://builds.apache.org/job/PreCommit-HDFS-Build/28310/testReport/] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14740) Recover data blocks from persistent memory read cache during datanode restarts
[
https://issues.apache.org/jira/browse/HDFS-14740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976700#comment-16976700
]
Rakesh Radhakrishnan commented on HDFS-14740:
-
Thanks [~Rui Mo] for the test result. Adding few comments, please address the
same. Apart from below comments, the patch looks good to me.
+Comment-1)+ Blocks will be persisted into the PMem device irrespective of the
config '{{dfs.datanode.cache.persistence.enabled}}' value, either true or
false. Again, the purpose of this flag is to restore the block cache in
datanode process using the physical blocks present in the PMem device. In that
sense how about renaming the config to
'{{dfs.datanode.cache.pmem.block.restore'}} or some other better name
reflecting the behavior?
+Comment-2)+ If not tested, then could you please capture the following
scenario into your test sheet.
*scenario:* User has cached {{file A}}. Now, admin has restarted datanode with
the above flag to {{false}}. Assume user has submitted cache directive command
to cache same {{file A}}.
> Recover data blocks from persistent memory read cache during datanode restarts
> --
>
> Key: HDFS-14740
> URL: https://issues.apache.org/jira/browse/HDFS-14740
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: caching, datanode
>Reporter: Feilong He
>Assignee: Rui Mo
>Priority: Major
> Attachments: HDFS-14740.000.patch, HDFS-14740.001.patch,
> HDFS-14740.002.patch, HDFS-14740.003.patch, HDFS-14740.004.patch,
> HDFS-14740.005.patch, HDFS-14740.006.patch,
> HDFS_Persistent_Read-Cache_Design-v1.pdf,
> HDFS_Persistent_Read-Cache_Test-v1.1.pdf,
> HDFS_Persistent_Read-Cache_Test-v1.pdf, HDFS_Persistent_Read-Cache_Test-v2.pdf
>
>
> In HDFS-13762, persistent memory (PM) is enabled in HDFS centralized cache
> management. Even though PM can persist cache data, for simplifying the
> initial implementation, the previous cache data will be cleaned up during
> DataNode restarts. Here, we are proposing to improve HDFS PM cache by taking
> advantage of PM's data persistence characteristic, i.e., recovering the
> status for cached data, if any, when DataNode restarts, thus, cache warm up
> time can be saved for user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2534) scmcli container delete not working
[
https://issues.apache.org/jira/browse/HDDS-2534?focusedWorklogId=345348=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345348
]
ASF GitHub Bot logged work on HDDS-2534:
Author: ASF GitHub Bot
Created on: 18/Nov/19 15:21
Start Date: 18/Nov/19 15:21
Worklog Time Spent: 10m
Work Description: chimney-lee commented on pull request #214:
HDDS-2534.scmcli container delete not working
URL: https://github.com/apache/hadoop-ozone/pull/214
## What changes were proposed in this pull request?
fix scmcli container delete not working
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-2534
## How was this patch tested?
exec `bin/ozone scmcli container delete `
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 345348)
Remaining Estimate: 0h
Time Spent: 10m
> scmcli container delete not working
> ---
>
> Key: HDDS-2534
> URL: https://issues.apache.org/jira/browse/HDDS-2534
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: luhuachao
>Assignee: luhuachao
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> java.lang.IllegalArgumentException: Unknown command type: DeleteContainer
> at
> org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.processRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:219)
> at
> org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72)
> at
> org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.submitRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:112)
> at
> org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java:30454)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2534) scmcli container delete not working
[
https://issues.apache.org/jira/browse/HDDS-2534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-2534:
-
Labels: pull-request-available (was: )
> scmcli container delete not working
> ---
>
> Key: HDDS-2534
> URL: https://issues.apache.org/jira/browse/HDDS-2534
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: luhuachao
>Assignee: luhuachao
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
>
> {code:java}
> java.lang.IllegalArgumentException: Unknown command type: DeleteContainer
> at
> org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.processRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:219)
> at
> org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72)
> at
> org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.submitRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:112)
> at
> org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java:30454)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14940) HDFS Balancer : getBalancerBandwidth displaying wrong values for the maximum network bandwidth used by the datanode while network bandwidth set with values as 104857600
[ https://issues.apache.org/jira/browse/HDFS-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976594#comment-16976594 ] hemanthboyina commented on HDFS-14940: -- yes you are correct , then we can have the validation check in distributed filesystem API . > HDFS Balancer : getBalancerBandwidth displaying wrong values for the maximum > network bandwidth used by the datanode while network bandwidth set with > values as 1048576000g/1048p/1e > --- > > Key: HDFS-14940 > URL: https://issues.apache.org/jira/browse/HDFS-14940 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer mover >Affects Versions: 3.1.1 > Environment: 3 Node HA Setup >Reporter: Souryakanta Dwivedy >Priority: Minor > Attachments: BalancerBW.PNG, HDFS-14940.001.patch > > > HDFS Balancer : getBalancerBandwidth displaying wrong values for the maximum > network bandwidth used by the datanode > while network bandwidth set with values as 1048576000g/1048p/1e > Steps :- > * Set balancer bandwith with command setBalancerBandwidth and vlaues as > [1048576000g/1048p/1e] > * - Check bandwidth used by the datanode during HDFS block balancing with > command :hdfs dfsadmin -getBalancerBandwidth " check it will display some > different values not the same value as set -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14651) DeadNodeDetector checks dead node periodically
[
https://issues.apache.org/jira/browse/HDFS-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976587#comment-16976587
]
Hadoop QA commented on HDFS-14651:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 15s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 1s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
30 unchanged - 0 fixed = 32 total (was 30) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
7s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}131m 2s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}226m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestDistributedFileSystemWithECFileWithRandomECPolicy |
| | hadoop.hdfs.TestEncryptedTransfer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14651 |
| JIRA Patch
[jira] [Updated] (HDFS-14992) TestOfflineEditsViewer is failing in Trunk
[ https://issues.apache.org/jira/browse/HDFS-14992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina updated HDFS-14992: - Description: [https://builds.apache.org/job/PreCommit-HDFS-Build/28310/testReport/] > TestOfflineEditsViewer is failing in Trunk > -- > > Key: HDFS-14992 > URL: https://issues.apache.org/jira/browse/HDFS-14992 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > > [https://builds.apache.org/job/PreCommit-HDFS-Build/28310/testReport/] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14992) TestOfflineEditsViewer is failing in Trunk
hemanthboyina created HDFS-14992: Summary: TestOfflineEditsViewer is failing in Trunk Key: HDFS-14992 URL: https://issues.apache.org/jira/browse/HDFS-14992 Project: Hadoop HDFS Issue Type: Bug Reporter: hemanthboyina Assignee: hemanthboyina -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
