[jira] [Commented] (HDFS-17001) Support getStatus API in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-17001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721220#comment-17721220 ] ASF GitHub Bot commented on HDFS-17001: --- slfan1989 commented on PR #5628: URL: https://github.com/apache/hadoop/pull/5628#issuecomment-1541472636 @zhtttylz Thanks for the contribution! We need to modify the code as suggested by @ayushtkn. > Support getStatus API in WebHDFS > > > Key: HDFS-17001 > URL: https://issues.apache.org/jira/browse/HDFS-17001 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.4.0 >Reporter: Hualong Zhang >Assignee: Hualong Zhang >Priority: Major > Labels: pull-request-available > Attachments: image-2023-05-08-14-34-51-873.png > > > WebHDFS should support getStatus: > !image-2023-05-08-14-34-51-873.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721235#comment-17721235
]
ASF GitHub Bot commented on HDFS-16979:
---
ayushtkn commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1189473819
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -462,13 +462,24 @@ private void appendClientPortToCallerContextIfAbsent() {
.setSignature(origSignature)
.build());
}
+ctx = CallerContext.getCurrent();
+if (isFormProxyUser(ctx)) {
+ CallerContext.setCurrent(
+ new CallerContext.Builder(ctx.getContext(), contextFieldSeparator)
+ .append(CallerContext.PROXY_USER_PORT,
String.valueOf(Server.getRemotePort()))
+ .setSignature(ctx.getSignature())
+ .build());
+}
}
private boolean isClientPortInfoAbsent(CallerContext ctx){
return ctx == null || ctx.getContext() == null
|| !ctx.getContext().contains(CallerContext.CLIENT_PORT_STR);
}
+ private boolean isFormProxyUser(CallerContext ctx) {
+return ctx != null &&
ctx.getContext().contains(CallerContext.REAL_USER_STR);
+ }
Review Comment:
typo: form -> from
Add a null check for getContext as well.
```
private boolean isFromProxyUser(CallerContext ctx) {
return ctx != null && ctx.getContext() != null && ctx.getContext()
.contains(CallerContext.REAL_USER_STR);
}
```
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java:
##
@@ -2116,6 +2121,8 @@ public void testRealUserPropagationInCallerContext()
// Real user is added to the caller context.
assertTrue("The audit log should contain the real user.",
logOutput.contains(String.format("realUser:%s",
realUser.getUserName(;
+assertTrue("The audit log should contain the proxyuser port.",
+logOutput.contains(PROXY_USER_PORT));
Review Comment:
can you assert the router port as well? There are 2 routers and it should be
either of them?
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17002) Erasure coding:Generate parity blocks in time to prevent file corruption
[ https://issues.apache.org/jira/browse/HDFS-17002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721241#comment-17721241 ] Stephen O'Donnell commented on HDFS-17002: -- If the directory scanner is not working, then the same issue can happen to any blocks on the system, replicated or EC if they are not read frequently. The system is designed to have the directory scanner running and its job is to detect corruptions such that you have described. I don't think there is any need to make any changes here. > Erasure coding:Generate parity blocks in time to prevent file corruption > > > Key: HDFS-17002 > URL: https://issues.apache.org/jira/browse/HDFS-17002 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.4.0 >Reporter: farmmamba >Priority: Major > > In current EC implementation, the corrupted parity block will not be > regenerated in time. > Think about below scene when using RS-6-3-1024k EC policy: > If three parity blocks p1, p2, p3 are all corrupted or deleted, we are not > aware of it. > Unfortunately, a data block is also corrupted in this time period, then this > file will be corrupted and can not be read by decoding. > > So, here we should always re-generate parity block in time when it is > unhealthy. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16965) Add switch to decide whether to enable native codec.
[
https://issues.apache.org/jira/browse/HDFS-16965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721245#comment-17721245
]
ASF GitHub Bot commented on HDFS-16965:
---
hadoop-yetus commented on PR #5520:
URL: https://github.com/apache/hadoop/pull/5520#issuecomment-1541562467
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 12s | | trunk passed |
| +1 :green_heart: | compile | 17m 11s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 15m 51s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 1m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 11s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 53s | | the patch passed |
| +1 :green_heart: | compile | 19m 17s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 19m 17s | | the patch passed |
| +1 :green_heart: | compile | 22m 18s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 22m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 10s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 41s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 44s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 56s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 27s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 20m 2s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not
generate ASF License warnings. |
| | | 200m 28s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5520/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux ece35d72c9d9 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c5174dd71b4ec5f1f0fdc7ad73030d41a48183e6 |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5520/5/testReport/ |
| Max. process+thread count | 1287 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5520/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://ye
[jira] [Commented] (HDFS-16985) delete local block file when FileNotFoundException occurred may lead to missing block.
[ https://issues.apache.org/jira/browse/HDFS-16985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721258#comment-17721258 ] ASF GitHub Bot commented on HDFS-16985: --- Hexiaoqiao commented on PR #5564: URL: https://github.com/apache/hadoop/pull/5564#issuecomment-1541600297 @jojochuang I will commit to trunk until wait two work days if no more comments. Thanks. > delete local block file when FileNotFoundException occurred may lead to > missing block. > -- > > Key: HDFS-16985 > URL: https://issues.apache.org/jira/browse/HDFS-16985 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Chengwei Wang >Assignee: Chengwei Wang >Priority: Major > Labels: pull-request-available > > We encounterd several missing-block problem in our production cluster which > hdfs running on AWS EC2 + EBS. > The root cause： > # the block remains only 1 replication left and hasn't been reconstruction > # DN checks block file existing when BlockSender construction > # the EBS checking failed and throw FileNotFoundException (EBS may be in > fault condition) > # DN invalidateBlock and schedule block async deletion > # EBS already back to normal when DN do delete block > # the block file be delete permanently and can't be recovered -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16965) Add switch to decide whether to enable native codec.
[
https://issues.apache.org/jira/browse/HDFS-16965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721271#comment-17721271
]
ASF GitHub Bot commented on HDFS-16965:
---
hadoop-yetus commented on PR #5520:
URL: https://github.com/apache/hadoop/pull/5520#issuecomment-1541763863
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 25s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 5s | | trunk passed |
| +1 :green_heart: | compile | 19m 41s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 22m 43s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 1m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 16s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 18m 2s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 18m 2s | | the patch passed |
| +1 :green_heart: | compile | 16m 0s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 16m 0s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 0s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 29s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 54s | | The patch does not
generate ASF License warnings. |
| | | 201m 22s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5520/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 5b4ed3e97a15 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / df3bda08b4681c9bcaa961efb90ef773b777dac8 |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5520/6/testReport/ |
| Max. process+thread count | 1261 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5520/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://ye
[jira] [Commented] (HDFS-17002) Erasure coding:Generate parity blocks in time to prevent file corruption
[ https://issues.apache.org/jira/browse/HDFS-17002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721273#comment-17721273 ] farmmamba commented on HDFS-17002: -- [~sodonnell] , thanks for your reply~ totally Agree with you. I will close this JIRA recently if no other friends discuss it. > Erasure coding:Generate parity blocks in time to prevent file corruption > > > Key: HDFS-17002 > URL: https://issues.apache.org/jira/browse/HDFS-17002 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.4.0 >Reporter: farmmamba >Priority: Major > > In current EC implementation, the corrupted parity block will not be > regenerated in time. > Think about below scene when using RS-6-3-1024k EC policy: > If three parity blocks p1, p2, p3 are all corrupted or deleted, we are not > aware of it. > Unfortunately, a data block is also corrupted in this time period, then this > file will be corrupted and can not be read by decoding. > > So, here we should always re-generate parity block in time when it is > unhealthy. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-17002) Erasure coding:Generate parity blocks in time to prevent file corruption
[ https://issues.apache.org/jira/browse/HDFS-17002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721273#comment-17721273 ] farmmamba edited comment on HDFS-17002 at 5/10/23 9:36 AM: --- [~sodonnell] , thanks for your reply~ totally agree with you. I will close this JIRA recently if no other friends discuss it. was (Author: zhanghaobo): [~sodonnell] , thanks for your reply~ totally Agree with you. I will close this JIRA recently if no other friends discuss it. > Erasure coding:Generate parity blocks in time to prevent file corruption > > > Key: HDFS-17002 > URL: https://issues.apache.org/jira/browse/HDFS-17002 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.4.0 >Reporter: farmmamba >Priority: Major > > In current EC implementation, the corrupted parity block will not be > regenerated in time. > Think about below scene when using RS-6-3-1024k EC policy: > If three parity blocks p1, p2, p3 are all corrupted or deleted, we are not > aware of it. > Unfortunately, a data block is also corrupted in this time period, then this > file will be corrupted and can not be read by decoding. > > So, here we should always re-generate parity block in time when it is > unhealthy. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16965) Add switch to decide whether to enable native codec.
[
https://issues.apache.org/jira/browse/HDFS-16965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721282#comment-17721282
]
ASF GitHub Bot commented on HDFS-16965:
---
YuanbenWang commented on code in PR #5520:
URL: https://github.com/apache/hadoop/pull/5520#discussion_r1189685731
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/CodecUtil.java:
##
@@ -78,6 +78,10 @@ public final class CodecUtil {
public static final String IO_ERASURECODE_CODEC_XOR_RAWCODERS_KEY =
IO_ERASURECODE_CODEC + "xor.rawcoders";
+ public static final String IO_ERASURECODE_CODEC_NATIVE_ENABLED_KEY =
"io.erasurecode.codec.native.enabled";
Review Comment:
@tomscut Thank you for your suggestion. I'v updated the code and could you
review again please?
> Add switch to decide whether to enable native codec.
>
>
> Key: HDFS-16965
> URL: https://issues.apache.org/jira/browse/HDFS-16965
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: erasure-coding
>Affects Versions: 3.3.4
>Reporter: WangYuanben
>Priority: Minor
> Labels: pull-request-available
>
> Sometimes we need to create codec without ISA-L, while priority is given to
> native codec by default. So it is necessary to add switch to decide whether
> to enable native codec.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721287#comment-17721287
]
ASF GitHub Bot commented on HDFS-16979:
---
LiuGuH commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1189714857
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java:
##
@@ -2116,6 +2121,8 @@ public void testRealUserPropagationInCallerContext()
// Real user is added to the caller context.
assertTrue("The audit log should contain the real user.",
logOutput.contains(String.format("realUser:%s",
realUser.getUserName(;
+assertTrue("The audit log should contain the proxyuser port.",
+logOutput.contains(PROXY_USER_PORT));
Review Comment:
Fixed. Thanks
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721288#comment-17721288
]
ASF GitHub Bot commented on HDFS-16979:
---
LiuGuH commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1189716104
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -462,13 +462,24 @@ private void appendClientPortToCallerContextIfAbsent() {
.setSignature(origSignature)
.build());
}
+ctx = CallerContext.getCurrent();
+if (isFormProxyUser(ctx)) {
+ CallerContext.setCurrent(
+ new CallerContext.Builder(ctx.getContext(), contextFieldSeparator)
+ .append(CallerContext.PROXY_USER_PORT,
String.valueOf(Server.getRemotePort()))
+ .setSignature(ctx.getSignature())
+ .build());
+}
}
private boolean isClientPortInfoAbsent(CallerContext ctx){
return ctx == null || ctx.getContext() == null
|| !ctx.getContext().contains(CallerContext.CLIENT_PORT_STR);
}
+ private boolean isFormProxyUser(CallerContext ctx) {
+return ctx != null &&
ctx.getContext().contains(CallerContext.REAL_USER_STR);
+ }
Review Comment:
Fixed. Thanks
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721289#comment-17721289
]
ASF GitHub Bot commented on HDFS-16979:
---
ayushtkn commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1189719329
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java:
##
@@ -2116,6 +2121,8 @@ public void testRealUserPropagationInCallerContext()
// Real user is added to the caller context.
assertTrue("The audit log should contain the real user.",
logOutput.contains(String.format("realUser:%s",
realUser.getUserName(;
+assertTrue("The audit log should contain the proxyuser port.",
+logOutput.contains(PROXY_USER_PORT));
Review Comment:
@LiuGuH did you miss this? checking for the actual port in the assert, if it
is the correct router port or not
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17000) Potential infinite loop in TestDFSStripedOutputStreamUpdatePipeline.testDFSStripedOutputStreamUpdatePipeline
[
https://issues.apache.org/jira/browse/HDFS-17000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721297#comment-17721297
]
Ayush Saxena commented on HDFS-17000:
-
yep, we should change int i to long.
Though technically it won't ever go that far, the code with go into the catch
block post 1024 * 1024 * 5, but makes sense to change to long.
[~Marcono1234] can you raise a PR to change to long. Let me know if you face
any issues. can try help
> Potential infinite loop in
> TestDFSStripedOutputStreamUpdatePipeline.testDFSStripedOutputStreamUpdatePipeline
>
>
> Key: HDFS-17000
> URL: https://issues.apache.org/jira/browse/HDFS-17000
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Marcono1234
>Priority: Major
>
> The method
> {{TestDFSStripedOutputStreamUpdatePipeline.testDFSStripedOutputStreamUpdatePipeline}}
> contains the following line:
> {code}
> for (int i = 0; i < Long.MAX_VALUE; i++) {
> {code}
> [GitHub source
> link|https://github.com/apache/hadoop/blob/4ee92efb73a90ae7f909e96de242d216ad6878b2/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamUpdatePipeline.java#L48]
> Because {{i}} is an {{int}} the condition {{i < Long.MAX_VALUE}} will always
> be true and {{i}} will simply overflow.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16984) Directory timestamp lost during the upgrade process
[
https://issues.apache.org/jira/browse/HDFS-16984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721299#comment-17721299
]
Ayush Saxena commented on HDFS-16984:
-
is it specific to one file? or happens with all files? do you plan to propose a
fix, if you have already found one
> Directory timestamp lost during the upgrade process
> ---
>
> Key: HDFS-16984
> URL: https://issues.apache.org/jira/browse/HDFS-16984
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.10.2, 3.3.5
>Reporter: Ke Han
>Priority: Major
> Attachments: GUBIkxOc.tar.gz
>
>
> h1. Symptoms
> The timestamp for a directory is lost after the upgrading from HDFS cluster
> 2.10.2 to 3.3.5.
> h1. Reproduce
> Start up a four-node HDFS cluster in 2.10.2 version.
> Execute the following commands. (The client is started up in NN, We have
> minimized the command sequence for reproducing)
> {code:java}
> bin/hdfs dfs -mkdir /GUBIkxOc
> bin/hdfs dfs -put -f -p -d /tmp/upfuzz/hdfs/GUBIkxOc/bQfxf /GUBIkxOc/
> bin/hdfs dfs -mkdir /GUBIkxOc/sKbTRjvS{code}
> Perform read in the old version
> {code:java}
> bin/hdfs dfs -ls -t -r -u /GUBIkxOc/
> Found 2 items
> drwxr-xr-x - root supergroup 0 1970-01-01 00:00 /GUBIkxOc/sKbTRjvS
> drwxr-xr-x - 20001 998 0 2023-04-17 16:15
> /GUBIkxOc/bQfxf{code}
>
> Then perform a full-stop upgrade to upgrade the entire cluster to 3.3.5.
> (Follow upgrade procedure in the website). When all nodes have started up, we
> perform the same read
> {code:java}
> Found 2 items
> drwxr-xr-x - 20001 998 0 1970-01-01 00:00 /GUBIkxOc/bQfxf
> drwxr-xr-x - root supergroup 0 1970-01-01 00:00
> /GUBIkxOc/sKbTRjvS {code}
> The timestamp info of directory /GUBIkxOc/bQfxf is lost. It changes from
> 2023-04-17 16:15 to 1970-01-01 00:00.
>
> I have also attached the required file: +/tmp/upfuzz/hdfs/GUBIkxOc/bQfxf+ .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16990) HttpFS Add Support getFileLinkStatus API
[
https://issues.apache.org/jira/browse/HDFS-16990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721311#comment-17721311
]
ASF GitHub Bot commented on HDFS-16990:
---
hadoop-yetus commented on PR #5602:
URL: https://github.com/apache/hadoop/pull/5602#issuecomment-1542057506
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 36s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 0m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 21s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 0m 49s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 5m 30s | | hadoop-hdfs-httpfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 100m 21s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5602/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5602 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5d3965259c4b 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5d794aac8b91ed7e9ada9b9075ed2d5a45092730 |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5602/4/testReport/ |
| Max. process+thread count | 819 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-httpfs U:
hadoop-hdfs-project/hadoop-hdfs-httpfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5602/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721315#comment-17721315
]
ASF GitHub Bot commented on HDFS-16979:
---
LiuGuH commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1189796280
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java:
##
@@ -2116,6 +2121,8 @@ public void testRealUserPropagationInCallerContext()
// Real user is added to the caller context.
assertTrue("The audit log should contain the real user.",
logOutput.contains(String.format("realUser:%s",
realUser.getUserName(;
+assertTrue("The audit log should contain the proxyuser port.",
+logOutput.contains(PROXY_USER_PORT));
Review Comment:
I saw it. It needs some time for me to achieve it. The correct router port
should get from correct Namenode Server.getRemotePort via the right rpc call.
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space
[
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721337#comment-17721337
]
ASF GitHub Bot commented on HDFS-3570:
--
ashutoshcipher commented on PR #5044:
URL: https://github.com/apache/hadoop/pull/5044#issuecomment-1542156086
@ZanderXu @slfan1989 any suggestions ?
> Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used
> space
>
>
> Key: HDFS-3570
> URL: https://issues.apache.org/jira/browse/HDFS-3570
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Affects Versions: 2.0.0-alpha
>Reporter: Harsh J
>Assignee: Ashutosh Gupta
>Priority: Minor
> Labels: pull-request-available
> Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch,
> HDFS-3570.aash.1.patch
>
>
> Report from a user here:
> https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
> post archived at http://pastebin.com/eVFkk0A0
> This user had a specific DN that had a large non-DFS usage among
> dfs.data.dirs, and very little DFS usage (which is computed against total
> possible capacity).
> Balancer apparently only looks at the usage, and ignores to consider that
> non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a
> DFS Usage report from DN is 8% only, its got a lot of free space to write
> more blocks, when that isn't true as shown by the case of this user. It went
> on scheduling writes to the DN to balance it out, but the DN simply can't
> accept any more blocks as a result of its disks' state.
> I think it would be better if we _computed_ the actual utilization based on
> {{(100-(actual remaining space))/(capacity)}}, as opposed to the current
> {{(dfs used)/(capacity)}}. Thoughts?
> This isn't very critical, however, cause it is very rare to see DN space
> being used for non DN data, but it does expose a valid bug.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Moved] (HDFS-17006) Compute correct checksum type when file is empty/ file length is 0
[ https://issues.apache.org/jira/browse/HDFS-17006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran moved HADOOP-18735 to HDFS-17006: Component/s: dfsclient (was: common) Key: HDFS-17006 (was: HADOOP-18735) Project: Hadoop HDFS (was: Hadoop Common) > Compute correct checksum type when file is empty/ file length is 0 > -- > > Key: HDFS-17006 > URL: https://issues.apache.org/jira/browse/HDFS-17006 > Project: Hadoop HDFS > Issue Type: Bug > Components: dfsclient >Reporter: Chayanika Bhandary >Priority: Trivial > Labels: pull-request-available > > When the file is empty or the file size is 0, the checksum returned is always > MD5MD5CRC type even when the selected checksum type is COMPOSITE_CRC. > This is misleading and can create confusion while debugging. > > Contributing a fix to this. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721482#comment-17721482
]
ASF GitHub Bot commented on HDFS-16979:
---
ayushtkn commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1190216959
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java:
##
@@ -2116,6 +2121,8 @@ public void testRealUserPropagationInCallerContext()
// Real user is added to the caller context.
assertTrue("The audit log should contain the real user.",
logOutput.contains(String.format("realUser:%s",
realUser.getUserName(;
+assertTrue("The audit log should contain the proxyuser port.",
+logOutput.contains(PROXY_USER_PORT));
Review Comment:
hmm, I think then we can live without it :)
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16978) RBF: Admin command to support bulk add of mount points
[ https://issues.apache.org/jira/browse/HDFS-16978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721483#comment-17721483 ] ASF GitHub Bot commented on HDFS-16978: --- virajjasani commented on PR #5554: URL: https://github.com/apache/hadoop/pull/5554#issuecomment-1542573847 Thanks again for the review @goiri, is this good for merge? i will create follow-up jiras to address: 1. rbf modules javadoc warnings for JDK 11 (these are real warnings that came into picture after we excluded proto package from maven-javadoc-plugin) 2. state store putAll implementations to also return entries that failed to get added/updated > RBF: Admin command to support bulk add of mount points > -- > > Key: HDFS-16978 > URL: https://issues.apache.org/jira/browse/HDFS-16978 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > > All state store implementations support adding multiple state store records > using single putAll() implementation. We should provide new router admin API > to support bulk addition of mount table entries that can utilize this build > add implementation at state store level. > For more than one mount point to be added, the goal of bulk addition should be > # To reduce frequent router calls > # To avoid frequent state store cache refreshers with each single mount > point addition -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16990) HttpFS Add Support getFileLinkStatus API
[ https://issues.apache.org/jira/browse/HDFS-16990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721485#comment-17721485 ] ASF GitHub Bot commented on HDFS-16990: --- ayushtkn merged PR #5602: URL: https://github.com/apache/hadoop/pull/5602 > HttpFS Add Support getFileLinkStatus API > > > Key: HDFS-16990 > URL: https://issues.apache.org/jira/browse/HDFS-16990 > Project: Hadoop HDFS > Issue Type: Improvement > Components: httpfs >Affects Versions: 3.4.0 >Reporter: Hualong Zhang >Assignee: Hualong Zhang >Priority: Major > Labels: pull-request-available > > HttpFS should implement the *getFileLinkStatus* API already implemented in > WebHDFS. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16990) HttpFS Add Support getFileLinkStatus API
[ https://issues.apache.org/jira/browse/HDFS-16990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721487#comment-17721487 ] Ayush Saxena commented on HDFS-16990: - Committed to trunk. Thanx [~zhtttylzz] for the contribution and [~slfan1989] for the review!!! > HttpFS Add Support getFileLinkStatus API > > > Key: HDFS-16990 > URL: https://issues.apache.org/jira/browse/HDFS-16990 > Project: Hadoop HDFS > Issue Type: Improvement > Components: httpfs >Affects Versions: 3.4.0 >Reporter: Hualong Zhang >Assignee: Hualong Zhang >Priority: Major > Labels: pull-request-available > > HttpFS should implement the *getFileLinkStatus* API already implemented in > WebHDFS. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16990) HttpFS Add Support getFileLinkStatus API
[ https://issues.apache.org/jira/browse/HDFS-16990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HDFS-16990. - Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > HttpFS Add Support getFileLinkStatus API > > > Key: HDFS-16990 > URL: https://issues.apache.org/jira/browse/HDFS-16990 > Project: Hadoop HDFS > Issue Type: Improvement > Components: httpfs >Affects Versions: 3.4.0 >Reporter: Hualong Zhang >Assignee: Hualong Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > HttpFS should implement the *getFileLinkStatus* API already implemented in > WebHDFS. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721507#comment-17721507
]
ASF GitHub Bot commented on HDFS-16979:
---
hadoop-yetus commented on PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#issuecomment-1542630798
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 37s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 55s | | trunk passed |
| +1 :green_heart: | compile | 17m 24s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 15m 41s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 4m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 4s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 29s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 7m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 4s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 32s | | the patch passed |
| +1 :green_heart: | compile | 16m 39s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 16m 39s | | the patch passed |
| +1 :green_heart: | compile | 15m 38s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 15m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 52s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/13/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 113 unchanged - 0 fixed = 114 total (was
113) |
| +1 :green_heart: | mvnsite | 3m 43s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 59s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 25s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 7m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 12s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 226m 14s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/13/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 21m 58s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/13/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 2s | | The patch does not
generate ASF License warnings. |
| | | 470m 49s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| |
hadoop.hdfs.server.federation.router.TestRouterRPCMultipleDestinationMountTableResolver
|
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/13/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle code
[jira] [Commented] (HDFS-17001) Support getStatus API in WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-17001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721552#comment-17721552
]
ASF GitHub Bot commented on HDFS-17001:
---
hadoop-yetus commented on PR #5628:
URL: https://github.com/apache/hadoop/pull/5628#issuecomment-1542838104
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 53s | | trunk passed |
| +1 :green_heart: | compile | 5m 13s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 5m 5s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 1m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 54s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 33s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 18s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 6m 55s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 18s | | the patch passed |
| +1 :green_heart: | compile | 5m 11s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 5m 11s | | the patch passed |
| +1 :green_heart: | compile | 4m 55s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 4m 55s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 6s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 2s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 2m 51s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 6m 53s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 20s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 21s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 208m 33s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5628/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 21m 7s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5628/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 52s | | The patch does not
generate ASF License warnings. |
| | | 370m 21s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| |
hadoop.hdfs.server.federation.router.TestRouterRPCMultipleDestinationMountTableResolver
|
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5628/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5628 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux f7b23d8406e0 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
19:10
[jira] [Commented] (HDFS-16978) RBF: Admin command to support bulk add of mount points
[ https://issues.apache.org/jira/browse/HDFS-16978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721562#comment-17721562 ] ASF GitHub Bot commented on HDFS-16978: --- goiri commented on PR #5554: URL: https://github.com/apache/hadoop/pull/5554#issuecomment-1542908688 LGTM. @ayushtkn can you give the final blessing? > RBF: Admin command to support bulk add of mount points > -- > > Key: HDFS-16978 > URL: https://issues.apache.org/jira/browse/HDFS-16978 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > > All state store implementations support adding multiple state store records > using single putAll() implementation. We should provide new router admin API > to support bulk addition of mount table entries that can utilize this build > add implementation at state store level. > For more than one mount point to be added, the goal of bulk addition should be > # To reduce frequent router calls > # To avoid frequent state store cache refreshers with each single mount > point addition -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16965) Add switch to decide whether to enable native codec.
[
https://issues.apache.org/jira/browse/HDFS-16965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721581#comment-17721581
]
ASF GitHub Bot commented on HDFS-16965:
---
tomscut commented on code in PR #5520:
URL: https://github.com/apache/hadoop/pull/5520#discussion_r1190559444
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/CodecUtil.java:
##
@@ -78,6 +78,10 @@ public final class CodecUtil {
public static final String IO_ERASURECODE_CODEC_XOR_RAWCODERS_KEY =
IO_ERASURECODE_CODEC + "xor.rawcoders";
+ public static final String IO_ERASURECODE_CODEC_NATIVE_ENABLED_KEY =
"io.erasurecode.codec.native.enabled";
Review Comment:
> @tomscut Thank you for your suggestion. I'v updated the code and could you
review again please?
Thanks. Now, it looks good to me.
> Add switch to decide whether to enable native codec.
>
>
> Key: HDFS-16965
> URL: https://issues.apache.org/jira/browse/HDFS-16965
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: erasure-coding
>Affects Versions: 3.3.4
>Reporter: WangYuanben
>Priority: Minor
> Labels: pull-request-available
>
> Sometimes we need to create codec without ISA-L, while priority is given to
> native codec by default. So it is necessary to add switch to decide whether
> to enable native codec.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17002) Erasure coding:Generate parity blocks in time to prevent file corruption
[ https://issues.apache.org/jira/browse/HDFS-17002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] farmmamba resolved HDFS-17002. -- Assignee: farmmamba Resolution: Not A Problem > Erasure coding:Generate parity blocks in time to prevent file corruption > > > Key: HDFS-17002 > URL: https://issues.apache.org/jira/browse/HDFS-17002 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > > In current EC implementation, the corrupted parity block will not be > regenerated in time. > Think about below scene when using RS-6-3-1024k EC policy: > If three parity blocks p1, p2, p3 are all corrupted or deleted, we are not > aware of it. > Unfortunately, a data block is also corrupted in this time period, then this > file will be corrupted and can not be read by decoding. > > So, here we should always re-generate parity block in time when it is > unhealthy. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17002) Erasure coding:Generate parity blocks in time to prevent file corruption
[ https://issues.apache.org/jira/browse/HDFS-17002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721582#comment-17721582 ] farmmamba commented on HDFS-17002: -- close this jira > Erasure coding:Generate parity blocks in time to prevent file corruption > > > Key: HDFS-17002 > URL: https://issues.apache.org/jira/browse/HDFS-17002 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > > In current EC implementation, the corrupted parity block will not be > regenerated in time. > Think about below scene when using RS-6-3-1024k EC policy: > If three parity blocks p1, p2, p3 are all corrupted or deleted, we are not > aware of it. > Unfortunately, a data block is also corrupted in this time period, then this > file will be corrupted and can not be read by decoding. > > So, here we should always re-generate parity block in time when it is > unhealthy. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16978) RBF: Admin command to support bulk add of mount points
[ https://issues.apache.org/jira/browse/HDFS-16978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721600#comment-17721600 ] ASF GitHub Bot commented on HDFS-16978: --- ayushtkn merged PR #5554: URL: https://github.com/apache/hadoop/pull/5554 > RBF: Admin command to support bulk add of mount points > -- > > Key: HDFS-16978 > URL: https://issues.apache.org/jira/browse/HDFS-16978 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > > All state store implementations support adding multiple state store records > using single putAll() implementation. We should provide new router admin API > to support bulk addition of mount table entries that can utilize this build > add implementation at state store level. > For more than one mount point to be added, the goal of bulk addition should be > # To reduce frequent router calls > # To avoid frequent state store cache refreshers with each single mount > point addition -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16978) RBF: Admin command to support bulk add of mount points
[ https://issues.apache.org/jira/browse/HDFS-16978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721601#comment-17721601 ] Ayush Saxena commented on HDFS-16978: - Committed to trunk. > RBF: Admin command to support bulk add of mount points > -- > > Key: HDFS-16978 > URL: https://issues.apache.org/jira/browse/HDFS-16978 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > > All state store implementations support adding multiple state store records > using single putAll() implementation. We should provide new router admin API > to support bulk addition of mount table entries that can utilize this build > add implementation at state store level. > For more than one mount point to be added, the goal of bulk addition should be > # To reduce frequent router calls > # To avoid frequent state store cache refreshers with each single mount > point addition -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16978) RBF: Admin command to support bulk add of mount points
[ https://issues.apache.org/jira/browse/HDFS-16978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HDFS-16978. - Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: Admin command to support bulk add of mount points > -- > > Key: HDFS-16978 > URL: https://issues.apache.org/jira/browse/HDFS-16978 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > All state store implementations support adding multiple state store records > using single putAll() implementation. We should provide new router admin API > to support bulk addition of mount table entries that can utilize this build > add implementation at state store level. > For more than one mount point to be added, the goal of bulk addition should be > # To reduce frequent router calls > # To avoid frequent state store cache refreshers with each single mount > point addition -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16978) RBF: Admin command to support bulk add of mount points
[ https://issues.apache.org/jira/browse/HDFS-16978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721602#comment-17721602 ] ASF GitHub Bot commented on HDFS-16978: --- ayushtkn commented on PR #5554: URL: https://github.com/apache/hadoop/pull/5554#issuecomment-1543273300 Done. thx everyone > RBF: Admin command to support bulk add of mount points > -- > > Key: HDFS-16978 > URL: https://issues.apache.org/jira/browse/HDFS-16978 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > All state store implementations support adding multiple state store records > using single putAll() implementation. We should provide new router admin API > to support bulk addition of mount table entries that can utilize this build > add implementation at state store level. > For more than one mount point to be added, the goal of bulk addition should be > # To reduce frequent router calls > # To avoid frequent state store cache refreshers with each single mount > point addition -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16965) Add switch to decide whether to enable native codec.
[
https://issues.apache.org/jira/browse/HDFS-16965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721611#comment-17721611
]
ASF GitHub Bot commented on HDFS-16965:
---
YuanbenWang commented on code in PR #5520:
URL: https://github.com/apache/hadoop/pull/5520#discussion_r1190603712
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/CodecUtil.java:
##
@@ -78,6 +78,10 @@ public final class CodecUtil {
public static final String IO_ERASURECODE_CODEC_XOR_RAWCODERS_KEY =
IO_ERASURECODE_CODEC + "xor.rawcoders";
+ public static final String IO_ERASURECODE_CODEC_NATIVE_ENABLED_KEY =
"io.erasurecode.codec.native.enabled";
Review Comment:
> > @tomscut Thank you for your suggestion. I'v updated the code and could
you review again please?
>
> Thanks. Now, it looks good to me.
Thank you for the review.
> Add switch to decide whether to enable native codec.
>
>
> Key: HDFS-16965
> URL: https://issues.apache.org/jira/browse/HDFS-16965
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: erasure-coding
>Affects Versions: 3.3.4
>Reporter: WangYuanben
>Priority: Minor
> Labels: pull-request-available
>
> Sometimes we need to create codec without ISA-L, while priority is given to
> native codec by default. So it is necessary to add switch to decide whether
> to enable native codec.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16978) RBF: Admin command to support bulk add of mount points
[ https://issues.apache.org/jira/browse/HDFS-16978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721626#comment-17721626 ] Viraj Jasani commented on HDFS-16978: - Thanks again [~ayushtkn] [~elgoiri] [~simbadzina] !!! > RBF: Admin command to support bulk add of mount points > -- > > Key: HDFS-16978 > URL: https://issues.apache.org/jira/browse/HDFS-16978 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > All state store implementations support adding multiple state store records > using single putAll() implementation. We should provide new router admin API > to support bulk addition of mount table entries that can utilize this build > add implementation at state store level. > For more than one mount point to be added, the goal of bulk addition should be > # To reduce frequent router calls > # To avoid frequent state store cache refreshers with each single mount > point addition -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16978) RBF: Admin command to support bulk add of mount points
[ https://issues.apache.org/jira/browse/HDFS-16978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721628#comment-17721628 ] Viraj Jasani commented on HDFS-16978: - will create follow-up jiras soon > RBF: Admin command to support bulk add of mount points > -- > > Key: HDFS-16978 > URL: https://issues.apache.org/jira/browse/HDFS-16978 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > All state store implementations support adding multiple state store records > using single putAll() implementation. We should provide new router admin API > to support bulk addition of mount table entries that can utilize this build > add implementation at state store level. > For more than one mount point to be added, the goal of bulk addition should be > # To reduce frequent router calls > # To avoid frequent state store cache refreshers with each single mount > point addition -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[ https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuguanghua updated HDFS-16979: --- Description: when client is using proxyuser via realuser, the hdfs aduilg log is lack of dfsrouter port infomation. client (using proxyuser)-> dfsrouter -> namenode clientport dfsrouterport hdfsauditlog should record dfsrouterport was: When remote client request through dfsrouter to namenode, the hdfsauditlog record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port. This patch is done for this scene. > RBF: Add dfsrouter port in hdfsauditlog > --- > > Key: HDFS-16979 > URL: https://issues.apache.org/jira/browse/HDFS-16979 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: liuguanghua >Priority: Major > Labels: pull-request-available > > > when client is using proxyuser via realuser, the hdfs aduilg log is lack of > dfsrouter port infomation. > client (using proxyuser)-> dfsrouter -> namenode > clientport dfsrouterport > hdfsauditlog should record dfsrouterport -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-17007) TestPendingReconstruction.testProcessPendingReconstructions verify HDFS-11960 test case is wrong
liuguanghua created HDFS-17007: -- Summary: TestPendingReconstruction.testProcessPendingReconstructions verify HDFS-11960 test case is wrong Key: HDFS-17007 URL: https://issues.apache.org/jira/browse/HDFS-17007 Project: Hadoop HDFS Issue Type: Bug Reporter: liuguanghua -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17007) TestPendingReconstruction.testProcessPendingReconstructions verify HDFS-11960 test case is wrong
[ https://issues.apache.org/jira/browse/HDFS-17007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuguanghua updated HDFS-17007: --- Description: Verify HDFS-11960 > TestPendingReconstruction.testProcessPendingReconstructions verify HDFS-11960 > test case is wrong > > > Key: HDFS-17007 > URL: https://issues.apache.org/jira/browse/HDFS-17007 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: liuguanghua >Priority: Minor > > > Verify HDFS-11960 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17007) TestPendingReconstruction.testProcessPendingReconstructions verify HDFS-11960 test case is wrong
[ https://issues.apache.org/jira/browse/HDFS-17007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuguanghua updated HDFS-17007: --- Description: TestPendingReconstruction.testProcessPendingReconstructions() verify HDFS-11960 is wrong. (1) It does not stop PendingReconstructionMonitor. The blockid will into timeouts queue because of timout duration is 3s. (2) Test blockid should be blk_1_1 with different genstamp. (3) The blk_1_1 should test with the same DatanodeDescriptor was: Verify HDFS-11960 > TestPendingReconstruction.testProcessPendingReconstructions verify HDFS-11960 > test case is wrong > > > Key: HDFS-17007 > URL: https://issues.apache.org/jira/browse/HDFS-17007 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: liuguanghua >Priority: Minor > > TestPendingReconstruction.testProcessPendingReconstructions() verify > HDFS-11960 is wrong. > (1) It does not stop PendingReconstructionMonitor. The blockid will into > timeouts queue because of timout duration is 3s. > (2) Test blockid should be blk_1_1 with different genstamp. > (3) The blk_1_1 should test with the same DatanodeDescriptor -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
