[jira] [Commented] (HDFS-14750) RBF: Improved isolation for downstream name nodes. {Dynamic}
[ https://issues.apache.org/jira/browse/HDFS-14750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619276#comment-17619276 ] JiangHua Zhu commented on HDFS-14750: - Thanks [~xuzq_zander] for the work. I have read your design and have some doubts: 1. Will the penalty time produced during the router run really disappear a lot? 2. Will it affect existing features such as Quota. 3. Overall, RBF still has a lot of room for development, and some compatibility needs to be considered. I have some ideas of my own that I can add to, if I can. 1: Add additional functions to each Router. include: 1.1. Collect your own processing performance indicators, similar to the role of sliding windows. 1.2. Dynamically set the maximum allowable processing upper limit according to the value of the sliding window. 2. Isolate the exception handler and namespace. Through these processes, the current problems can be effectively alleviated, and good compatibility can also be maintained. > RBF: Improved isolation for downstream name nodes. {Dynamic} > > > Key: HDFS-14750 > URL: https://issues.apache.org/jira/browse/HDFS-14750 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Labels: pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > > This Jira tracks the work around dynamic allocation of resources in routers > for downstream hdfs clusters. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16749) RBF: Gets the wrong directory information from Trash
[ https://issues.apache.org/jira/browse/HDFS-16749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiping Zhang reassigned HDFS-16749: --- Assignee: Xiping Zhang > RBF: Gets the wrong directory information from Trash > > > Key: HDFS-16749 > URL: https://issues.apache.org/jira/browse/HDFS-16749 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.4.0 >Reporter: Xiping Zhang >Assignee: Xiping Zhang >Priority: Critical > > // Current User :zhangxiping > // router =》 ns0 ,ns1 > {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfsrouteradmin -ls{color} > Mount Table Entries: > Source Destinations Owner > Group Mode Quota/Usage > /user/a ns0->/user/a(Priority:0,Readonly:false) zhangxiping > Administrators rwxr-xr-x [NsQuota: -/-, SsQuota: > -/-] > /user/b ns0->/user/b(Priority:0,Readonly:false) zhangxiping > Administrators rwxr-xr-x [NsQuota: -/-, SsQuota: > -/-] > /user/c ns0->/user/c(Priority:0,Readonly:false) zhangxiping > Administrators rwxr-xr-x [NsQuota: -/-, SsQuota: > -/-] > {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfs -ls hdfs://ns0/user{color} > Found 4 items > drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:17 > hdfs://ns0/user/a > drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:17 > hdfs://ns0/user/b > drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:17 > hdfs://ns0/user/c > drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:11 > hdfs://ns0/user/zhangxiping > {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfs -ls hdfs://ns1/user{color} > ls: `hdfs://ns1/user': No such file or directory > {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfs -ls > hdfs://ns0/user/zhangxiping/.Trash/Current/user{color} > Found 1 items > drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:14 > hdfs://ns0/user/zhangxiping/.Trash/Current/user/d > {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfs -ls > /user/zhangxiping/.Trash/Current/user {color:#505f79}//Requests to the > router{color}{color} > Found 4 items > {color:#00875a}drwxrwxrwx - zhangxiping supergroup 0 1970-01-01 > 08:00 /user/zhangxiping/.Trash/Current/user/a{color} > {color:#00875a}drwxrwxrwx - zhangxiping supergroup 0 1970-01-01 > 08:00 /user/zhangxiping/.Trash/Current/user/b{color} > {color:#00875a}drwxrwxrwx - zhangxiping supergroup 0 1970-01-01 > 08:00 /user/zhangxiping/.Trash/Current/user/c *//The records on it > shouldn't exist*{color} > drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:14 > /user/zhangxiping/.Trash/Current/user/d -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16716) Improve appendToFile command: support appending on file with new block
[ https://issues.apache.org/jira/browse/HDFS-16716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619318#comment-17619318 ] ASF GitHub Bot commented on HDFS-16716: --- liuhongtong commented on PR #4697: URL: https://github.com/apache/hadoop/pull/4697#issuecomment-1281968645 Very useful feature. Merged into the version used internally, although the community has not yet merged. > Improve appendToFile command: support appending on file with new block > -- > > Key: HDFS-16716 > URL: https://issues.apache.org/jira/browse/HDFS-16716 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: guojunhao >Priority: Minor > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > HDFS client DistributedFileSystem#append supports appending to a file with > optional create flags. > However, appendToFile command only supports the default create flag APPEND so > that append on EC file without NEW_BLOCK create flag is not supported. > Thus, it's necessary to improve appendToFile command by adding option n for > it. Option n represents that use NEW_BLOCK create flag while appending file. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16749) RBF: Gets the wrong directory information from Trash
[ https://issues.apache.org/jira/browse/HDFS-16749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619352#comment-17619352 ] ASF GitHub Bot commented on HDFS-16749: --- hadoop-yetus commented on PR #5039: URL: https://github.com/apache/hadoop/pull/5039#issuecomment-1282026050 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 1m 28s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 1s | | codespell was not available. | | +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 42m 11s | | trunk passed | | +1 :green_heart: | compile | 0m 55s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | | +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 0m 41s | | trunk passed | | +1 :green_heart: | mvnsite | 0m 52s | | trunk passed | | +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | | +1 :green_heart: | javadoc | 1m 6s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 1m 42s | | trunk passed | | +1 :green_heart: | shadedclient | 24m 15s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 42s | | the patch passed | | +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | | +1 :green_heart: | javac | 0m 41s | | the patch passed | | +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 0m 36s | | the patch passed | | +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 0m 20s | | the patch passed | | +1 :green_heart: | mvnsite | 0m 39s | | the patch passed | | +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | | +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | spotbugs | 1m 26s | | the patch passed | | +1 :green_heart: | shadedclient | 26m 22s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 36m 46s | | hadoop-hdfs-rbf in the patch passed. | | +1 :green_heart: | asflicense | 0m 46s | | The patch does not generate ASF License warnings. | | | | 146m 36s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5039/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/5039 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets | | uname | Linux 1e7895ce4003 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 69fc8ffebd241bba5797b6d466b4cf566b09e2b6 | | Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5039/1/testReport/ | | Max. process+thread count | 3589 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5039/1/console | | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > RBF: Gets the w
[jira] [Updated] (HDFS-16749) RBF: Gets the wrong directory information from Trash
[ https://issues.apache.org/jira/browse/HDFS-16749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16749: -- Labels: pull-request-available (was: ) > RBF: Gets the wrong directory information from Trash > > > Key: HDFS-16749 > URL: https://issues.apache.org/jira/browse/HDFS-16749 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.4.0 >Reporter: Xiping Zhang >Assignee: Xiping Zhang >Priority: Critical > Labels: pull-request-available > > // Current User :zhangxiping > // router =》 ns0 ,ns1 > {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfsrouteradmin -ls{color} > Mount Table Entries: > Source Destinations Owner > Group Mode Quota/Usage > /user/a ns0->/user/a(Priority:0,Readonly:false) zhangxiping > Administrators rwxr-xr-x [NsQuota: -/-, SsQuota: > -/-] > /user/b ns0->/user/b(Priority:0,Readonly:false) zhangxiping > Administrators rwxr-xr-x [NsQuota: -/-, SsQuota: > -/-] > /user/c ns0->/user/c(Priority:0,Readonly:false) zhangxiping > Administrators rwxr-xr-x [NsQuota: -/-, SsQuota: > -/-] > {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfs -ls hdfs://ns0/user{color} > Found 4 items > drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:17 > hdfs://ns0/user/a > drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:17 > hdfs://ns0/user/b > drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:17 > hdfs://ns0/user/c > drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:11 > hdfs://ns0/user/zhangxiping > {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfs -ls hdfs://ns1/user{color} > ls: `hdfs://ns1/user': No such file or directory > {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfs -ls > hdfs://ns0/user/zhangxiping/.Trash/Current/user{color} > Found 1 items > drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:14 > hdfs://ns0/user/zhangxiping/.Trash/Current/user/d > {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfs -ls > /user/zhangxiping/.Trash/Current/user {color:#505f79}//Requests to the > router{color}{color} > Found 4 items > {color:#00875a}drwxrwxrwx - zhangxiping supergroup 0 1970-01-01 > 08:00 /user/zhangxiping/.Trash/Current/user/a{color} > {color:#00875a}drwxrwxrwx - zhangxiping supergroup 0 1970-01-01 > 08:00 /user/zhangxiping/.Trash/Current/user/b{color} > {color:#00875a}drwxrwxrwx - zhangxiping supergroup 0 1970-01-01 > 08:00 /user/zhangxiping/.Trash/Current/user/c *//The records on it > shouldn't exist*{color} > drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:14 > /user/zhangxiping/.Trash/Current/user/d -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16805) Support set the number of RPC Readers according to different RPC Servers in NameNode
[ https://issues.apache.org/jira/browse/HDFS-16805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619359#comment-17619359 ] Haiyang Hu commented on HDFS-16805: --- [~jianghuazhu] Thanks, let me look > Support set the number of RPC Readers according to different RPC Servers in > NameNode > > > Key: HDFS-16805 > URL: https://issues.apache.org/jira/browse/HDFS-16805 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > > Currently, multiple rpc servers are started in namenode, such as client rpc > server, service rpc server, lifeline rpc server, and each rpc server used the > same parameter 'ipc.server.read.threadpool.size' setting for the number of > reader threads for this server . > Consider according to different rpc server requirements set the number of > reader threads for this server,for example: > In client RPC server use parameter 'dfs.namenode.reader.count' , > In service RPC server use parameter 'dfs.namenode.service.reader.count' , > In lifeline RPC server use parameter 'dfs.namenode.lifeline.reader.count' -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16749) RBF: Gets the wrong directory information from Trash
[ https://issues.apache.org/jira/browse/HDFS-16749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiping Zhang updated HDFS-16749: Description: *Liststatus trash path, some directory information that does not exist is returned. Look at the following example:* // Current User :zhangxiping // router =》 ns0 ,ns1 {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfsrouteradmin -ls{color} Mount Table Entries: Source Destinations Owner Group Mode Quota/Usage /user/a ns0->/user/a(Priority:0,Readonly:false) zhangxiping Administrators rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user/b ns0->/user/b(Priority:0,Readonly:false) zhangxiping Administrators rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user/c ns0->/user/c(Priority:0,Readonly:false) zhangxiping Administrators rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfs -ls hdfs://ns0/user{color} Found 4 items drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:17 hdfs://ns0/user/a drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:17 hdfs://ns0/user/b drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:17 hdfs://ns0/user/c drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:11 hdfs://ns0/user/zhangxiping {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfs -ls hdfs://ns1/user{color} ls: `hdfs://ns1/user': No such file or directory {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfs -ls hdfs://ns0/user/zhangxiping/.Trash/Current/user{color} Found 1 items drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:14 hdfs://ns0/user/zhangxiping/.Trash/Current/user/d {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfs -ls /user/zhangxiping/.Trash/Current/user //Requests to the router{color} Found 4 items drwxrwxrwx - zhangxiping supergroup 0 1970-01-01 08:00 /user/zhangxiping/.Trash/Current/user/a {color:#00875a}drwxrwxrwx - zhangxiping supergroup 0 1970-01-01 08:00 /user/zhangxiping/.Trash/Current/user/b{color} {color:#00875a}drwxrwxrwx - zhangxiping supergroup 0 1970-01-01 08:00 /user/zhangxiping/.Trash/Current/user/c *//The records on it shouldn't exist*{color} drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:14 /user/zhangxiping/.Trash/Current/user/d was: // Current User :zhangxiping // router =》 ns0 ,ns1 {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfsrouteradmin -ls{color} Mount Table Entries: Source Destinations Owner Group Mode Quota/Usage /user/a ns0->/user/a(Priority:0,Readonly:false) zhangxiping Administrators rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user/b ns0->/user/b(Priority:0,Readonly:false) zhangxiping Administrators rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user/c ns0->/user/c(Priority:0,Readonly:false) zhangxiping Administrators rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfs -ls hdfs://ns0/user{color} Found 4 items drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:17 hdfs://ns0/user/a drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:17 hdfs://ns0/user/b drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:17 hdfs://ns0/user/c drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:11 hdfs://ns0/user/zhangxiping {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfs -ls hdfs://ns1/user{color} ls: `hdfs://ns1/user': No such file or directory {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfs -ls hdfs://ns0/user/zhangxiping/.Trash/Current/user{color} Found 1 items drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:14 hdfs://ns0/user/zhangxiping/.Trash/Current/user/d {*}\project\hadoop3.3.0{*}>{color:#ff}hdfs dfs -ls /user/zhangxiping/.Trash/Current/user {color:#505f79}//Requests to the router{color}{color} Found 4 items {color:#00875a}drwxrwxrwx - zhangxiping supergroup 0 1970-01-01 08:00 /user/zhangxiping/.Trash/Current/user/a{color} {color:#00875a}drwxrwxrwx - zhangxiping supergroup 0 1970-01-01 08:00 /user/zhangxiping/.Trash/Current/user/b{color} {color:#00875a}drwxrwxrwx - zhangxiping supergroup 0 1970-01-01 08:00 /user/zhangxiping/.Trash/Current/user/c *//The records on it shouldn't exist*{color} drwxr-xr-x - zhangxiping supergroup 0 2022-10-18 10:14 /user/zhangxiping/.Trash/Current/user/d > RBF: Gets the wrong directory information from Trash > > > Key: HDFS-16749