[jira] [Commented] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[ https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16455222#comment-16455222 ] Hudson commented on HDFS-12370: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14070 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14070/]) HDFS-12370. Ozone: Implement TopN container choosing policy for (omalley: rev fec04f11587fdbe5f77425444e2453149afb7d9b) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/ContainerManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDeletionChoosingPolicy.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerPersistence.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/impl/TopNOrderedContainerDeletionChoosingPolicy.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/ozone-default.xml * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerStatus.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/DeleteBlocksCommandHandler.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/background/BlockDeletingService.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/impl/RandomContainerDeletionChoosingPolicy.java > Ozone: Implement TopN container choosing policy for BlockDeletionService > > > Key: HDFS-12370 > URL: https://issues.apache.org/jira/browse/HDFS-12370 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Major > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12370-HDFS-7240.001.patch, > HDFS-12370-HDFS-7240.002.patch, HDFS-12370-HDFS-7240.003.patch > > > Implement TopN container choosing policy for BlockDeletionService. This is > discussed from HDFS-12354. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[ https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16450987#comment-16450987 ] Hudson commented on HDFS-12370: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14057 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14057/]) HDFS-12370. Ozone: Implement TopN container choosing policy for (wwei: rev bbea878c7b8db2d2f28970129b532f81d0615c7c) * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/impl/TopNOrderedContainerDeletionChoosingPolicy.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerStatus.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/impl/RandomContainerDeletionChoosingPolicy.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/ContainerManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDeletionChoosingPolicy.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/background/BlockDeletingService.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerManagerImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerPersistence.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/DeleteBlocksCommandHandler.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/ozone-default.xml > Ozone: Implement TopN container choosing policy for BlockDeletionService > > > Key: HDFS-12370 > URL: https://issues.apache.org/jira/browse/HDFS-12370 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Major > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12370-HDFS-7240.001.patch, > HDFS-12370-HDFS-7240.002.patch, HDFS-12370-HDFS-7240.003.patch > > > Implement TopN container choosing policy for BlockDeletionService. This is > discussed from HDFS-12354. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[ https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16164307#comment-16164307 ] Yiqun Lin commented on HDFS-12370: -- Thanks for providing reference, [~cheersyang]. Maybe we should adjust parameters value in current deletion services. I will take some time to look into this. > Ozone: Implement TopN container choosing policy for BlockDeletionService > > > Key: HDFS-12370 > URL: https://issues.apache.org/jira/browse/HDFS-12370 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12370-HDFS-7240.001.patch, > HDFS-12370-HDFS-7240.002.patch, HDFS-12370-HDFS-7240.003.patch > > > Implement TopN container choosing policy for BlockDeletionService. This is > discussed from HDFS-12354. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[ https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161170#comment-16161170 ] Weiwei Yang commented on HDFS-12370: Hi [~linyiqun] Thanks for sharing your idea. The problem here is that different layers are using different message format to transfer deletion messages, so it is not easy to define a single throttle mechanism for all these places. You can find more details in the up-to-date [^document|https://issues.apache.org/jira/secure/attachment/12886359/Asynchronous%20key%20delete%20.pdf] I uploaded in HDFS-11922, please take a look at the document and let me know if you have any good idea to resolve this, thanks! > Ozone: Implement TopN container choosing policy for BlockDeletionService > > > Key: HDFS-12370 > URL: https://issues.apache.org/jira/browse/HDFS-12370 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12370-HDFS-7240.001.patch, > HDFS-12370-HDFS-7240.002.patch, HDFS-12370-HDFS-7240.003.patch > > > Implement TopN container choosing policy for BlockDeletionService. This is > discussed from HDFS-12354. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[ https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161098#comment-16161098 ] Yiqun Lin commented on HDFS-12370: -- Thanks for the review and commit, [~cheersyang].I noticed you mentioned one point that current deletion behaviour is really slow. bq. I think that was because the throttle logic at KSM or SCM side were non-optimal Now we did the throttle logic across many phases. Can we throttle it only in source place by default? I mean that we just do this in KSM since deletion operation is invoked fistly in KSM side. Since the source place is throttled, I think there will also not be much pending deletion blocks that to be deleted afterward. Users can enable throttle in SCM if they want. > Ozone: Implement TopN container choosing policy for BlockDeletionService > > > Key: HDFS-12370 > URL: https://issues.apache.org/jira/browse/HDFS-12370 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12370-HDFS-7240.001.patch, > HDFS-12370-HDFS-7240.002.patch, HDFS-12370-HDFS-7240.003.patch > > > Implement TopN container choosing policy for BlockDeletionService. This is > discussed from HDFS-12354. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[
https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161004#comment-16161004
]
Weiwei Yang commented on HDFS-12370:
I have tested this patch in a cluster, here is the steps I have tried
1. Decrease the container size from 5G to 1MB, this way we can create a lot of
containers for testing.
2. Create 2000 keys using corona
{code}
./bin/hdfs corona -numOfThreads 10 -numOfVolumes 1 -numOfBuckets 1 -numOfKeys
2000
{code}
3. Check containers have been created, there were 20
{code}
[wwei@ozone1 containers]$ ls
1d163265-8d47-4ed3-845f-f7d3eb569b83 b0865025-acfe-48a0-a5b2-d36224ecdb20
21017018-be32-44e6-9f62-e7fc9f6a6021 bcdb31bb-beee-47d0-8734-e407e64789c5
4cccadc8-ef5e-466d-bd9e-5b9705f8748c c7a85e6e-3528-45d1-8063-cd7fec114545
55b11be9-f16f-4620-a310-07c9be3bbfee ce883b83-2543-4b81-bdaf-6813b817b889
684556ea-a668-4bd3-82ee-8836db6fcba5 d7e3a36c-c68b-454c-80ce-23e3e92f3b1a
73e1a9b5-b77a-4230-a148-86b7f945dd63 da94ea5d-a279-4955-9c09-726579a167ec
75885243-3ae7-4f55-82dc-475f3d98f1f3 e753d4e6-19a1-4344-a97c-853d63ca0f81
77d3a401-8ece-41f0-b8e6-88d071aee301 e98ccb38-6b1e-4dc7-ac15-0f985f678e1f
84fdce99-3f69-4600-a16c-fc4fee20da01 ece9d60a-d5c9-434c-95bf-b3c425693eca
8f888499-397a-4c24-86bd-18dc73d2f870 ff1075d6-37d2-4b70-aae9-9cb58839f9ea
{code}
4. Get all keys by listVolume, listBucket and listKey calls, and redirect all
keys to a json file
{code}
./bin/hdfs oz -listVolume http://localhost:9864 -user wwei
./bin/hdfs oz -listBucket http://localhost:9864/vol-0-70079 -user wwei
./bin/hdfs oz -listKey http://localhost:9864/vol-0-70079/bucket-0-62742 -user
wwei > /tmp/keys.json
{code}
5. Write a python script to parse {{keys.json}} and delete keys one by one
{code}
import json
import subprocess
with open('keys.json') as data_file:
data = json.load(data_file)
for key in data:
print key['keyName']
fullName="http://localhost:9864/vol-0-70079/bucket-0-62742/"+key['keyName']
print "deleting key " + fullName
subprocess.check_output(["bash",
"/home/wwei/hadoop-3.0.0-beta1-SNAPSHOT/bin/hdfs", "oz", "-deleteKey",
fullName, "-user", "wwei"])
{code}
6. From the log, it seems it scans containers according to the order of the
number of pending deletion keys.
{code}
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container e98ccb38-6b1e-4dc7-ac15-0f985f678e1f for block deletion, pending
deletion blocks num: 9.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion, pending
deletion blocks num: 1.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 21017018-be32-44e6-9f62-e7fc9f6a6021 for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 4cccadc8-ef5e-466d-bd9e-5b9705f8748c for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 55b11be9-f16f-4620-a310-07c9be3bbfee for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 684556ea-a668-4bd3-82ee-8836db6fcba5 for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 73e1a9b5-b77a-4230-a148-86b7f945dd63 for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 75885243-3ae7-4f55-82dc-475f3d98f1f3 for block deletion, pending
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 77d3a401-8ece-41f0-b8e6-88d071aee301 for block deletion, pending
deletion blocks num: 0.
...
17/09/11 03:00:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 77d3a401-8ece-41f0-b8e6-88d071aee301 for block deletion, pending
deletion blocks num: 4.
17/09/11 03:00:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container e98ccb38-6b1e-4dc7-ac15-0f985f678e1f for block deletion, pending
deletion blocks num: 4.
17/09/11 03:00:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion, pending
deletion blocks num: 1.
17/09/11 03:00:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion, pending
deletion blocks num: 0.
17/09/11 03:00:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select
container 21017018-be32-44e6-9f62-e7f
[jira] [Commented] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[
https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150475#comment-16150475
]
Hadoop QA commented on HDFS-12370:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
45s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 36s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}104m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12370 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884932/HDFS-12370-HDFS-7240.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux c4c9403de366 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 3c8f1c5 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20966/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20966/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20966/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: Implement TopN container choosing policy for BlockDeletionService
>
>
> Key: HDFS-12370
> URL: https://issues.apache.org/jira/browse/HDFS
[jira] [Commented] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[
https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150368#comment-16150368
]
Yiqun Lin commented on HDFS-12370:
--
Seems pending deletion blocks initialize logic moved into {{readContainerInfo}}
broken the test {{TestContainerPersistence}}.
Attach the new patch to fix failure unit test and fix remaining checkstyle
warnings.
> Ozone: Implement TopN container choosing policy for BlockDeletionService
>
>
> Key: HDFS-12370
> URL: https://issues.apache.org/jira/browse/HDFS-12370
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-12370-HDFS-7240.001.patch,
> HDFS-12370-HDFS-7240.002.patch, HDFS-12370-HDFS-7240.003.patch
>
>
> Implement TopN container choosing policy for BlockDeletionService. This is
> discussed from HDFS-12354.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[ https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150305#comment-16150305 ] Weiwei Yang commented on HDFS-12370: Hi [~linyiqun] v2 patch looks good to me, can you fix the remaining checkstyle issues please. Thanks. > Ozone: Implement TopN container choosing policy for BlockDeletionService > > > Key: HDFS-12370 > URL: https://issues.apache.org/jira/browse/HDFS-12370 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-12370-HDFS-7240.001.patch, > HDFS-12370-HDFS-7240.002.patch > > > Implement TopN container choosing policy for BlockDeletionService. This is > discussed from HDFS-12354. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[
https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150270#comment-16150270
]
Hadoop QA commented on HDFS-12370:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
32s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 38s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 1 unchanged - 0 fixed = 5 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 49s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}113m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.ozone.container.common.impl.TestContainerPersistence |
| | hadoop.ozone.web.client.TestKeys |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12370 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884896/HDFS-12370-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 9968ed3d21d9 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 3c8f1c5 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/20964/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20964/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20964/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
ha
[jira] [Commented] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[ https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150164#comment-16150164 ] Yiqun Lin commented on HDFS-12370: -- Thanks [~cheersyang] for the review, all the comments make sense to me. Attach the updated patch. > Ozone: Implement TopN container choosing policy for BlockDeletionService > > > Key: HDFS-12370 > URL: https://issues.apache.org/jira/browse/HDFS-12370 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-12370-HDFS-7240.001.patch, > HDFS-12370-HDFS-7240.002.patch > > > Implement TopN container choosing policy for BlockDeletionService. This is > discussed from HDFS-12354. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[
https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148709#comment-16148709
]
Weiwei Yang commented on HDFS-12370:
Hi [~linyiqun]
Thanks for uploading the patch, it looks good to me overall. Some minor comments
*ContainerManagerImpl.java*
# line 185 - 193, can we move the init code to
{{ContainerManagerImpl#readContainerInfo}} ?
# method {{incrNumDeletionBlocks}}, {{decrNumDeletionBlocks}}, why not use
{{incrPendingDeletionBlocks}} and {{decrPendingDeletionBlocks}} as well? Which
seems more accurate to what it does.
*ContainerStatus.java*
# It seems this class is not thread safe, but it will need to be called in
multiple threads in {{BlockDeletingService}}
# line 79 and 88, there seems a small typo, {{Deletin}} -> {{Deletion}}
# line 93 I guess you overrides {{compareTo}} so you can directly use
Collection.sort to sort a ContainerStatus list, but it doesn't seem to be good
to use number of pending deletion blocks to differentiate two ContainerStatus
instances. Can you pass a customized comparator instead? Such as
{{Collections.sort(orderedList, new Comparator ...);}}
*TopNOrderedContainerDeletionChoosingPolicy.java*
NIT, can we add a log message after line 54 to print how many pending deletion
blocks this container has ? This will help to illustrate how it works.
Thank you.
> Ozone: Implement TopN container choosing policy for BlockDeletionService
>
>
> Key: HDFS-12370
> URL: https://issues.apache.org/jira/browse/HDFS-12370
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-12370-HDFS-7240.001.patch
>
>
> Implement TopN container choosing policy for BlockDeletionService. This is
> discussed from HDFS-12354.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[
https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148502#comment-16148502
]
Hadoop QA commented on HDFS-12370:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
29s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
29s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 40s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | org.apache.hadoop.ozone.container.common.impl.ContainerStatus defines
compareTo(ContainerStatus) and uses Object.equals() At
ContainerStatus.java:Object.equals() At ContainerStatus.java:[line 94] |
| Failed junit tests | hadoop.ozone.web.client.TestKeys |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12370 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884586/HDFS-12370-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 404239d0793c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / b23c267 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/20933/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/20933/artifact/patchproces
[jira] [Commented] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[ https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148428#comment-16148428 ] Yiqun Lin commented on HDFS-12370: -- Attach initial patch. > Ozone: Implement TopN container choosing policy for BlockDeletionService > > > Key: HDFS-12370 > URL: https://issues.apache.org/jira/browse/HDFS-12370 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > > Implement TopN container choosing policy for BlockDeletionService. This is > discussed from HDFS-12354. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12370) Ozone: Implement TopN container choosing policy for BlockDeletionService
[ https://issues.apache.org/jira/browse/HDFS-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144981#comment-16144981 ] Weiwei Yang commented on HDFS-12370: Thanks for filing this task, look forward for a patch. For those who may not have full context, Yiqun and I had some discussion in HDFS-12354 on this, see [this comment|https://issues.apache.org/jira/browse/HDFS-12354?focusedCommentId=16143430&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16143430]. > Ozone: Implement TopN container choosing policy for BlockDeletionService > > > Key: HDFS-12370 > URL: https://issues.apache.org/jira/browse/HDFS-12370 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > > Implement TopN container choosing policy for BlockDeletionService. This is > discussed from HDFS-12354. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
