[jira] [Updated] (HDFS-16822) HostRestrictingAuthorizationFilter should pass through requests if they don't access WebHDFS API
[ https://issues.apache.org/jira/browse/HDFS-16822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-16822: Description: After HDFS-15320, HostRestrictingAuthorizationFilter returns 404 error if it receives a request that doesn't access WebHDFS API. With this change, the endpoints, such as /conf and /jmx are no longer visible. This is very inconvenient for administrators. HostRestrictingAuthorizationFilter should pass through requests if they don't access WebHDFS API. This issue is reported by [~hadachi]. was: After HDFS-15320, HostRestrictingAuthorizationFilter returns 404 error if it receives a request that doesn't access WebHDFS API. With this change, the endpoints, such as /conf and /jmx are no longer visible. This is very inconvenient for administrators. HostRestrictingAuthorizationFilter should pass through requests if they don't access WebHDFS API. > HostRestrictingAuthorizationFilter should pass through requests if they don't > access WebHDFS API > > > Key: HDFS-16822 > URL: https://issues.apache.org/jira/browse/HDFS-16822 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > > After HDFS-15320, HostRestrictingAuthorizationFilter returns 404 error if it > receives a request that doesn't access WebHDFS API. > With this change, the endpoints, such as /conf and /jmx are no longer > visible. This is very inconvenient for administrators. > HostRestrictingAuthorizationFilter should pass through requests if they don't > access WebHDFS API. > This issue is reported by [~hadachi]. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16822) HostRestrictingAuthorizationFilter should pass through requests if they don't access WebHDFS API
[ https://issues.apache.org/jira/browse/HDFS-16822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16822: -- Labels: pull-request-available (was: ) > HostRestrictingAuthorizationFilter should pass through requests if they don't > access WebHDFS API > > > Key: HDFS-16822 > URL: https://issues.apache.org/jira/browse/HDFS-16822 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > > After HDFS-15320, HostRestrictingAuthorizationFilter returns 404 error if it > receives a request that doesn't access WebHDFS API. > With this change, the endpoints, such as /conf and /jmx are no longer > visible. This is very inconvenient for administrators. > HostRestrictingAuthorizationFilter should pass through requests if they don't > access WebHDFS API. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16822) HostRestrictingAuthorizationFilter should pass through requests if they don't access WebHDFS API
[ https://issues.apache.org/jira/browse/HDFS-16822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624200#comment-17624200 ] ASF GitHub Bot commented on HDFS-16822: --- tasanuma opened a new pull request, #5079: URL: https://github.com/apache/hadoop/pull/5079 … ### Description of PR After HDFS-15320, HostRestrictingAuthorizationFilter returns 404 error if it receives a request that doesn't access WebHDFS API. With this change, the endpoints, such as /conf and /jmx are no longer visible. This is very inconvenient for administrators. HostRestrictingAuthorizationFilter should pass through requests if they don't access WebHDFS API. ### How was this patch tested? - verify the unit test - verify it in my local cluster ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? > HostRestrictingAuthorizationFilter should pass through requests if they don't > access WebHDFS API > > > Key: HDFS-16822 > URL: https://issues.apache.org/jira/browse/HDFS-16822 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > > After HDFS-15320, HostRestrictingAuthorizationFilter returns 404 error if it > receives a request that doesn't access WebHDFS API. > With this change, the endpoints, such as /conf and /jmx are no longer > visible. This is very inconvenient for administrators. > HostRestrictingAuthorizationFilter should pass through requests if they don't > access WebHDFS API. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16822) HostRestrictingAuthorizationFilter should pass through requests if they don't access WebHDFS API
Takanobu Asanuma created HDFS-16822: --- Summary: HostRestrictingAuthorizationFilter should pass through requests if they don't access WebHDFS API Key: HDFS-16822 URL: https://issues.apache.org/jira/browse/HDFS-16822 Project: Hadoop HDFS Issue Type: Task Reporter: Takanobu Asanuma Assignee: Takanobu Asanuma After HDFS-15320, HostRestrictingAuthorizationFilter returns 404 error if it receives a request that doesn't access WebHDFS API. With this change, the endpoints, such as /conf and /jmx are no longer visible. This is very inconvenient for administrators. HostRestrictingAuthorizationFilter should pass through requests if they don't access WebHDFS API. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16821) Fix regression in HDFS-13522 that enables observer reads by default.
[ https://issues.apache.org/jira/browse/HDFS-16821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624183#comment-17624183 ] ASF GitHub Bot commented on HDFS-16821: --- simbadzina opened a new pull request, #5078: URL: https://github.com/apache/hadoop/pull/5078 ### Fixes regression in HDFS-13522 that enables observer reads by default ### Refactored existing unit tests and added a new one **testReadWithoutObserverClientConfigurations** that failing before code change but not passes with the regression fixed. ### For code changes: - [ x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? > Fix regression in HDFS-13522 that enables observer reads by default. > > > Key: HDFS-16821 > URL: https://issues.apache.org/jira/browse/HDFS-16821 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Major > > Serving reads consistently from Observer Namenodes is a feature that was > introduced in HDFS-12943. > Clients opt-into this feature by configuring the ObserverReadProxyProvider. > It is important that the opt-in is explicit because for third-party reads to > remain consistent, these clients then need to perform an msync before reads. > In HDFS-13522, the ClientGSIContext is implicitly added to the DFSClient thus > enabling Observer reads for all clients by default. This breaks consistency > guarantees for clients that haven't opted into observer reads. > [https://github.com/apache/hadoop/pull/4883/files#diff-a627e2c1f3e68235520d3c28092f4ae8a41aa4557cc530e4e6862c318be7e898R352-R354] > We need to return to the old behavior of only using the ClientGSIContext when > users have explicitly opted into Observer reads. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16821) Fix regression in HDFS-13522 that enables observer reads by default.
[ https://issues.apache.org/jira/browse/HDFS-16821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16821: -- Labels: pull-request-available (was: ) > Fix regression in HDFS-13522 that enables observer reads by default. > > > Key: HDFS-16821 > URL: https://issues.apache.org/jira/browse/HDFS-16821 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Major > Labels: pull-request-available > > Serving reads consistently from Observer Namenodes is a feature that was > introduced in HDFS-12943. > Clients opt-into this feature by configuring the ObserverReadProxyProvider. > It is important that the opt-in is explicit because for third-party reads to > remain consistent, these clients then need to perform an msync before reads. > In HDFS-13522, the ClientGSIContext is implicitly added to the DFSClient thus > enabling Observer reads for all clients by default. This breaks consistency > guarantees for clients that haven't opted into observer reads. > [https://github.com/apache/hadoop/pull/4883/files#diff-a627e2c1f3e68235520d3c28092f4ae8a41aa4557cc530e4e6862c318be7e898R352-R354] > We need to return to the old behavior of only using the ClientGSIContext when > users have explicitly opted into Observer reads. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16821) Fix regression in HDFS-13522 that enables observer reads by default.
[ https://issues.apache.org/jira/browse/HDFS-16821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simbarashe Dzinamarira updated HDFS-16821: -- Summary: Fix regression in HDFS-13522 that enables observer reads by default. (was: HDFS clients must not use ClientGSIContext unless explicit configured to do so.) > Fix regression in HDFS-13522 that enables observer reads by default. > > > Key: HDFS-16821 > URL: https://issues.apache.org/jira/browse/HDFS-16821 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Major > > Serving reads consistently from Observer Namenodes is a feature that was > introduced in HDFS-12943. > Clients opt-into this feature by configuring the ObserverReadProxyProvider. > It is important that the opt-in is explicit because for third-party reads to > remain consistent, these clients then need to perform an msync before reads. > In HDFS-13522, the ClientGSIContext is implicitly added to the DFSClient thus > enabling Observer reads for all clients by default. This breaks consistency > guarantees for clients that haven't opted into observer reads. > [https://github.com/apache/hadoop/pull/4883/files#diff-a627e2c1f3e68235520d3c28092f4ae8a41aa4557cc530e4e6862c318be7e898R352-R354] > We need to return to the old behavior of only using the ClientGSIContext when > users have explicitly opted into Observer reads. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16821) HDFS clients must not use ClientGSIContext unless explicit configured to do so.
[ https://issues.apache.org/jira/browse/HDFS-16821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simbarashe Dzinamarira reassigned HDFS-16821: - Assignee: Simbarashe Dzinamarira > HDFS clients must not use ClientGSIContext unless explicit configured to do > so. > --- > > Key: HDFS-16821 > URL: https://issues.apache.org/jira/browse/HDFS-16821 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Major > > Serving reads consistently from Observer Namenodes is a feature that was > introduced in HDFS-12943. > Clients opt-into this feature by configuring the ObserverReadProxyProvider. > It is important that the opt-in is explicit because for third-party reads to > remain consistent, these clients then need to perform an msync before reads. > In HDFS-13522, the ClientGSIContext is implicitly added to the DFSClient thus > enabling Observer reads for all clients by default. This breaks consistency > guarantees for clients that haven't opted into observer reads. > [https://github.com/apache/hadoop/pull/4883/files#diff-a627e2c1f3e68235520d3c28092f4ae8a41aa4557cc530e4e6862c318be7e898R352-R354] > We need to return to the old behavior of only using the ClientGSIContext when > users have explicitly opted into Observer reads. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16821) HDFS clients must not use ClientGSIContext unless explicit configured to do so.
Simbarashe Dzinamarira created HDFS-16821: - Summary: HDFS clients must not use ClientGSIContext unless explicit configured to do so. Key: HDFS-16821 URL: https://issues.apache.org/jira/browse/HDFS-16821 Project: Hadoop HDFS Issue Type: Bug Components: hdfs Reporter: Simbarashe Dzinamarira Serving reads consistently from Observer Namenodes is a feature that was introduced in HDFS-12943. Clients opt-into this feature by configuring the ObserverReadProxyProvider. It is important that the opt-in is explicit because for third-party reads to remain consistent, these clients then need to perform an msync before reads. In HDFS-13522, the ClientGSIContext is implicitly added to the DFSClient thus enabling Observer reads for all clients by default. This breaks consistency guarantees for clients that haven't opted into observer reads. [https://github.com/apache/hadoop/pull/4883/files#diff-a627e2c1f3e68235520d3c28092f4ae8a41aa4557cc530e4e6862c318be7e898R352-R354] We need to return to the old behavior of only using the ClientGSIContext when users have explicitly opted into Observer reads. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[ https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624178#comment-17624178 ] ASF GitHub Bot commented on HDFS-11242: --- secfree commented on PR #3790: URL: https://github.com/apache/hadoop/pull/3790#issuecomment-1291490442 > hi, @secfree , why was this PR closed ? @hfutatzhanghb I closed it because no one feedback/review in 6 months.The PR itself works well in our clusters. > Add refresh cluster network topology operation to dfs admin > --- > > Key: HDFS-11242 > URL: https://issues.apache.org/jira/browse/HDFS-11242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Reid Chan >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-11242.002.patch, HDFS-11242.patch > > Time Spent: 50m > Remaining Estimate: 0h > > The network topology and dns to switch mapping are initialized at the start > of the namenode. > If admin wants to change the topology because of new datanodes added, he has > to stop and restart namenode(s), otherwise those new added datanodes are > squeezed under /default-rack. > It is a low frequency operation, but it should be operated appropriately, so > dfs admin should take the responsibility. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16816) RBF: auto-create user home dir for trash paths by router
[
https://issues.apache.org/jira/browse/HDFS-16816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624173#comment-17624173
]
ASF GitHub Bot commented on HDFS-16816:
---
hadoop-yetus commented on PR #5071:
URL: https://github.com/apache/hadoop/pull/5071#issuecomment-1291473368
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 16s | | trunk passed |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 38s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 5s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 35m 23s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 141m 10s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5071/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5071 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 174626bea195 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1de60c62ba7199d53f603eb10f7e33cb4fb72cf7 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5071/6/testReport/ |
| Max. process+thread count | 3175 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5071/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> RBF: auto-create user home dir
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[ https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624170#comment-17624170 ] ASF GitHub Bot commented on HDFS-11242: --- hfutatzhanghb commented on PR #3790: URL: https://github.com/apache/hadoop/pull/3790#issuecomment-1291465337 hi, @secfree , why was this PR closed ? > Add refresh cluster network topology operation to dfs admin > --- > > Key: HDFS-11242 > URL: https://issues.apache.org/jira/browse/HDFS-11242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Reid Chan >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-11242.002.patch, HDFS-11242.patch > > Time Spent: 50m > Remaining Estimate: 0h > > The network topology and dns to switch mapping are initialized at the start > of the namenode. > If admin wants to change the topology because of new datanodes added, he has > to stop and restart namenode(s), otherwise those new added datanodes are > squeezed under /default-rack. > It is a low frequency operation, but it should be operated appropriately, so > dfs admin should take the responsibility. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16716) Improve appendToFile command: support appending on file with new block
[
https://issues.apache.org/jira/browse/HDFS-16716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624157#comment-17624157
]
ASF GitHub Bot commented on HDFS-16716:
---
M1eyu2018 commented on code in PR #4697:
URL: https://github.com/apache/hadoop/pull/4697#discussion_r1005170440
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:
##
@@ -3043,6 +3044,101 @@ public void testAppendToFileBadArgs() throws Exception {
assertThat(res, not(0));
}
+ @Test (timeout = 30)
+ public void testAppendToFileWithOptionN() throws Exception {
+final int inputFileLength = 1024 * 1024;
+File testRoot = new File(TEST_ROOT_DIR, "testAppendToFileWithOptionN");
+testRoot.mkdirs();
+
+File file1 = new File(testRoot, "file1");
+createLocalFileWithRandomData(inputFileLength, file1);
+
+Configuration conf = new HdfsConfiguration();
+MiniDFSCluster cluster = new
MiniDFSCluster.Builder(conf).numDataNodes(6).build();
+cluster.waitActive();
+
+try {
Review Comment:
Thanks @tomscut for your correction. I have updated the branch.
> Improve appendToFile command: support appending on file with new block
> --
>
> Key: HDFS-16716
> URL: https://issues.apache.org/jira/browse/HDFS-16716
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: guojunhao
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> HDFS client DistributedFileSystem#append supports appending to a file with
> optional create flags.
> However, appendToFile command only supports the default create flag APPEND so
> that append on EC file without NEW_BLOCK create flag is not supported.
> Thus, it's necessary to improve appendToFile command by adding option n for
> it. Option n represents that use NEW_BLOCK create flag while appending file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16820) Document Fix for hadoop.user.group.metrics.percentiles.intervals
teng wang created HDFS-16820:
Summary: Document Fix for
hadoop.user.group.metrics.percentiles.intervals
Key: HDFS-16820
URL: https://issues.apache.org/jira/browse/HDFS-16820
Project: Hadoop HDFS
Issue Type: Bug
Components: configuration
Affects Versions: 3.3.4
Reporter: teng wang
The default value of hadoop.user.group.metrics.percentiles.intervals is
dfs.metrics.percentiles.intervals. If
hadoop.user.group.metrics.percentiles.intervals is not set, it would take the
value of dfs.metrics.percentiles.intervals.
It is suggested to fix the document of hdfs-default.xml
{code:java}
//
hadoop.user.group.metrics.percentiles.intervals
A comma-separated list of the granularity in seconds for the metrics
which describe the 50/75/90/95/99th percentile latency for group resolution
in milliseconds.
By default, percentile latency metrics are disabled.
+ If not specified, the value of dfs.metrics.percentiles.intervals is used
as the default value.
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16813) Remove parameter validation logic such as dfs.namenode.decommission.blocks.per.interval in DatanodeAdminManager#activate
[
https://issues.apache.org/jira/browse/HDFS-16813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624140#comment-17624140
]
ASF GitHub Bot commented on HDFS-16813:
---
haiyang1987 commented on PR #5063:
URL: https://github.com/apache/hadoop/pull/5063#issuecomment-1291413789
Update PR. Hi @ZanderXu @goiri can you help to review again, Thanks.
> Remove parameter validation logic such as
> dfs.namenode.decommission.blocks.per.interval in DatanodeAdminManager#activate
>
>
> Key: HDFS-16813
> URL: https://issues.apache.org/jira/browse/HDFS-16813
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> In DatanodeAdminManager#activate
> {code:java}
> int blocksPerInterval = conf.getInt(
> DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_KEY,
> DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_DEFAULT);
> final String deprecatedKey =
> "dfs.namenode.decommission.nodes.per.interval";
> final String strNodes = conf.get(deprecatedKey);
> if (strNodes != null) {
> LOG.warn("Deprecated configuration key {} will be ignored.",
> deprecatedKey);
> LOG.warn("Please update your configuration to use {} instead.",
> DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_KEY);
> }
> checkArgument(blocksPerInterval > 0,
> "Must set a positive value for "
> + DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_KEY);
> final int maxConcurrentTrackedNodes = conf.getInt(
> DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_MAX_CONCURRENT_TRACKED_NODES,
> DFSConfigKeys
> .DFS_NAMENODE_DECOMMISSION_MAX_CONCURRENT_TRACKED_NODES_DEFAULT);
> checkArgument(maxConcurrentTrackedNodes >= 0, "Cannot set a negative " +
> "value for "
> + DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_MAX_CONCURRENT_TRACKED_NODES);
> {code}
> there is no need for parameters
> dfs.namenode.decommission.blocks.per.interval and
> dfs.namenode.decommission.max.concurrent.tracked.nodes to verify.
> Because the parameters are processed in DatanodeAdminMonitorBase and
> DatanodeAdminDefaultMonitor.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16802) Print options when accessing ClientProtocol#rename2()
[
https://issues.apache.org/jira/browse/HDFS-16802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624133#comment-17624133
]
ASF GitHub Bot commented on HDFS-16802:
---
ZanderXu commented on code in PR #5013:
URL: https://github.com/apache/hadoop/pull/5013#discussion_r1005141101
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java:
##
@@ -247,10 +247,8 @@ static RenameResult renameToInt(
throws IOException {
String src = srcArg;
String dst = dstArg;
-if (NameNode.stateChangeLog.isDebugEnabled()) {
- NameNode.stateChangeLog.debug("DIR* NameSystem.renameTo: with options -"
+
- " " + src + " to " + dst);
-}
+NameNode.stateChangeLog.debug("DIR* NameSystem.renameTo: with options={}
{} to {}",
+Arrays.toString(options), src, dst);
Review Comment:
Oh, here is my mistake. @jianghuazhu we need `if
(NameNode.stateChangeLog.isDebugEnabled())` here, because
`Arrays.toString(options)` will cost some time to compose a string.
> Print options when accessing ClientProtocol#rename2()
> -
>
> Key: HDFS-16802
> URL: https://issues.apache.org/jira/browse/HDFS-16802
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.3.4
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
> Labels: pull-request-available
>
> When accessing ClientProtocol#rename2(), the carried options cannot be seen
> in the log. Here is some log information:
> {code:java}
> 2022-10-13 10:21:10,727 [Listener at localhost/59732] DEBUG hdfs.StateChange
> (FSDirRenameOp.java:renameToInt(255)) - DIR* NameSystem.renameTo: with
> options - /testNamenodeRetryCache/testRename2/src to
> /testNamenodeRetryCache/testRename2/target
> {code}
> We should improve this, maybe printing options would be better.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13791) Limit logging frequency of edit tail related statements
[
https://issues.apache.org/jira/browse/HDFS-13791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624119#comment-17624119
]
Chengbing Liu commented on HDFS-13791:
--
In our production cluster with Observer NameNode enabled, we have plenty of
logs printed by {{FSEditLogLoader}} and {{RedundantEditLogInputStream}}. The
{{LogThrottlingHelper}} doesn't seem to work.
{noformat}
2022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
Start loading edits file ByteStringEditLog[17686250688, 17686250688],
ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688,
17686250688] maxTxnsToRead = 9223372036854775807
2022-10-25 09:26:50,380 INFO
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream:
Fast-forwarding stream 'ByteStringEditLog[17686250688, 17686250688],
ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688,
17686250688]' to transaction ID 17686250688
2022-10-25 09:26:50,380 INFO
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream:
Fast-forwarding stream 'ByteStringEditLog[17686250688, 17686250688]' to
transaction ID 17686250688
2022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
Loaded 1 edits file(s) (the last named ByteStringEditLog[17686250688,
17686250688], ByteStringEditLog[17686250688, 17686250688],
ByteStringEditLog[17686250688, 17686250688]) of total size 527.0, total edits
1.0, total load time 0.0 ms
2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
Start loading edits file ByteStringEditLog[17686250689, 17686250693],
ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689,
17686250693] maxTxnsToRead = 9223372036854775807
2022-10-25 09:26:50,387 INFO
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream:
Fast-forwarding stream 'ByteStringEditLog[17686250689, 17686250693],
ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689,
17686250693]' to transaction ID 17686250689
2022-10-25 09:26:50,387 INFO
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream:
Fast-forwarding stream 'ByteStringEditLog[17686250689, 17686250693]' to
transaction ID 17686250689
2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
Loaded 1 edits file(s) (the last named ByteStringEditLog[17686250689,
17686250693], ByteStringEditLog[17686250689, 17686250693],
ByteStringEditLog[17686250689, 17686250693]) of total size 890.0, total edits
5.0, total load time 1.0 ms
{noformat}
After some digging, I found the cause is that {{LogThrottlingHelper}}'s are
declared as instance variables of all the enclosing classes, including
{{FSImage}}, {{FSEditLogLoader}} and {{RedundantEditLogInputStream}}. Therefore
the logging frequency will not be limited across different instances. For
classes with only limited number of instances, such as {{FSImage}}, this is
fine. For others whose instances will be created continuously, such as
{{FSEditLogLoader}} and {{RedundantEditLogInputStream}}, it will result in
plenty of logs.
[~xkrogen] How about making them static variables?
> Limit logging frequency of edit tail related statements
> ---
>
> Key: HDFS-13791
> URL: https://issues.apache.org/jira/browse/HDFS-13791
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs, qjm
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: HDFS-12943, 3.3.0
>
> Attachments: HDFS-13791-HDFS-12943.000.patch,
> HDFS-13791-HDFS-12943.001.patch, HDFS-13791-HDFS-12943.002.patch,
> HDFS-13791-HDFS-12943.003.patch, HDFS-13791-HDFS-12943.004.patch,
> HDFS-13791-HDFS-12943.005.patch, HDFS-13791-HDFS-12943.006.patch
>
>
> There are a number of log statements that occur every time new edits are
> tailed by a Standby NameNode. When edits are tailing only on the order of
> every tens of seconds, this is fine. With the work in HDFS-13150, however,
> edits may be tailed every few milliseconds, which can flood the logs with
> tailing-related statements. We should throttle it to limit it to printing at
> most, say, once per 5 seconds.
> We can implement logic similar to that used in HDFS-10713. This may be
> slightly more tricky since the log statements are distributed across a few
> classes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16816) RBF: auto-create user home dir for trash paths by router
[
https://issues.apache.org/jira/browse/HDFS-16816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624116#comment-17624116
]
ASF GitHub Bot commented on HDFS-16816:
---
hadoop-yetus commented on PR #5071:
URL: https://github.com/apache/hadoop/pull/5071#issuecomment-1291348446
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 17s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 23s | | trunk passed |
| +1 :green_heart: | compile | 0m 51s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 48s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 59s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 6s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 27s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 29s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5071/5/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 4
unchanged - 0 fixed = 5 total (was 4) |
| +1 :green_heart: | mvnsite | 0m 42s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 38m 8s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 153m 16s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5071/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5071 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux dae824abd7c5 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2503f7bb54236e0ac37db93d942c27d5261b396e |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5071/5/testReport/ |
| Max. process+thread count | 3407 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
[jira] [Commented] (HDFS-16550) [SBN read] Improper cache-size for journal node may cause cluster crash
[
https://issues.apache.org/jira/browse/HDFS-16550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624115#comment-17624115
]
ASF GitHub Bot commented on HDFS-16550:
---
tomscut commented on PR #4209:
URL: https://github.com/apache/hadoop/pull/4209#issuecomment-1291347727
> @tomscut Thanks for involving me. In my case, I think this PR is
unnecessary. But we can print some warning logs to prompt the admin if the set
memory is too large, such as more than 90% of the heap size.
>
> But, if anyone thinks this modification is necessary, I will review it
carefully later.
Thanks @ZanderXu for the review. There are already waring logs, but they are
easy to ignore. Because there is no connection between memory and cache size,
it's easy to miss when updating configuration.
> [SBN read] Improper cache-size for journal node may cause cluster crash
> ---
>
> Key: HDFS-16550
> URL: https://issues.apache.org/jira/browse/HDFS-16550
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Tao Li
>Assignee: Tao Li
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2022-04-21-09-54-29-751.png,
> image-2022-04-21-09-54-57-111.png, image-2022-04-21-12-32-56-170.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> When we introduced {*}SBN Read{*}, we encountered a situation during upgrade
> the JournalNodes.
> Cluster Info:
> *Active: nn0*
> *Standby: nn1*
> 1. Rolling restart journal node. {color:#ff}(related config:
> fs.journalnode.edit-cache-size.bytes=1G, -Xms1G, -Xmx=1G){color}
> 2. The cluster runs for a while, edits cache usage is increasing and memory
> is used up.
> 3. {color:#ff}Active namenode(nn0){color} shutdown because of “{_}Timed
> out waiting 12ms for a quorum of nodes to respond”{_}.
> 4. Transfer nn1 to Active state.
> 5. {color:#ff}New Active namenode(nn1){color} also shutdown because of
> “{_}Timed out waiting 12ms for a quorum of nodes to respond” too{_}.
> 6. {color:#ff}The cluster crashed{color}.
>
> Related code:
> {code:java}
> JournaledEditsCache(Configuration conf) {
> capacity = conf.getInt(DFSConfigKeys.DFS_JOURNALNODE_EDIT_CACHE_SIZE_KEY,
> DFSConfigKeys.DFS_JOURNALNODE_EDIT_CACHE_SIZE_DEFAULT);
> if (capacity > 0.9 * Runtime.getRuntime().maxMemory()) {
> Journal.LOG.warn(String.format("Cache capacity is set at %d bytes but " +
> "maximum JVM memory is only %d bytes. It is recommended that you " +
> "decrease the cache size or increase the heap size.",
> capacity, Runtime.getRuntime().maxMemory()));
> }
> Journal.LOG.info("Enabling the journaled edits cache with a capacity " +
> "of bytes: " + capacity);
> ReadWriteLock lock = new ReentrantReadWriteLock(true);
> readLock = new AutoCloseableLock(lock.readLock());
> writeLock = new AutoCloseableLock(lock.writeLock());
> initialize(INVALID_TXN_ID);
> } {code}
> Currently, *fs.journalNode.edit-cache-size-bytes* can be set to a larger size
> than the memory requested by the process. If
> {*}fs.journalNode.edit-cache-sie.bytes > 0.9 *
> Runtime.getruntime().maxMemory(){*}, only warn logs are printed during
> journalnode startup. This can easily be overlooked by users. However, as the
> cluster runs to a certain period of time, it is likely to cause the cluster
> to crash.
>
> NN log:
> !image-2022-04-21-09-54-57-111.png|width=1012,height=47!
> !image-2022-04-21-12-32-56-170.png|width=809,height=218!
> IMO, we should not set the {{cache size}} to a fixed value, but to the ratio
> of maximum memory, which is 0.2 by default.
> This avoids the problem of too large cache size. In addition, users can
> actively adjust the heap size when they need to increase the cache size.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16716) Improve appendToFile command: support appending on file with new block
[
https://issues.apache.org/jira/browse/HDFS-16716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624113#comment-17624113
]
ASF GitHub Bot commented on HDFS-16716:
---
tomscut commented on code in PR #4697:
URL: https://github.com/apache/hadoop/pull/4697#discussion_r1005123746
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java:
##
@@ -1544,6 +1544,39 @@ public FSDataOutputStream append(Path f, int bufferSize)
throws IOException {
public abstract FSDataOutputStream append(Path f, int bufferSize,
Progressable progress) throws IOException;
+ /**
+ * Append to an existing file (optional operation).
+ * @param f the existing file to be appended.
+ * @param appendToNewBlock whether to append data to a new block
+ * instead of the end of the last partial block
+ * @throws IOException IO failure
+ * @throws UnsupportedOperationException if the operation is unsupported
+ * (default).
+ * @return output stream.
+ */
+ public FSDataOutputStream append(Path f, boolean appendToNewBlock) throws
IOException {
+return append(f, getConf().getInt(IO_FILE_BUFFER_SIZE_KEY,
+IO_FILE_BUFFER_SIZE_DEFAULT), null, appendToNewBlock);
+ }
+
+ /**
+ * Append to an existing file (optional operation).
+ * This function is used for being overridden by some FileSystem like
DistributedFileSystem
+ * @param f the existing file to be appended.
+ * @param bufferSize the size of the buffer to be used.
+ * @param progress for reporting progress if it is not null.
+ * @param appendToNewBlock whether to append data to a new block
+ * instead of the end of the last partial block
+ * @throws IOException IO failure
+ * @throws UnsupportedOperationException if the operation is unsupported
+ * (default).
+ * @return output stream.
+ */
+ public FSDataOutputStream append(Path f, int bufferSize,
+ Progressable progress, boolean appendToNewBlock) throws IOException {
+return append(f, bufferSize, progress);
Review Comment:
Hi @ZanderXu , do you have any other suggestions? Thank you.
> Improve appendToFile command: support appending on file with new block
> --
>
> Key: HDFS-16716
> URL: https://issues.apache.org/jira/browse/HDFS-16716
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: guojunhao
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> HDFS client DistributedFileSystem#append supports appending to a file with
> optional create flags.
> However, appendToFile command only supports the default create flag APPEND so
> that append on EC file without NEW_BLOCK create flag is not supported.
> Thus, it's necessary to improve appendToFile command by adding option n for
> it. Option n represents that use NEW_BLOCK create flag while appending file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16716) Improve appendToFile command: support appending on file with new block
[
https://issues.apache.org/jira/browse/HDFS-16716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624110#comment-17624110
]
ASF GitHub Bot commented on HDFS-16716:
---
tomscut commented on code in PR #4697:
URL: https://github.com/apache/hadoop/pull/4697#discussion_r1005123080
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:
##
@@ -3043,6 +3044,101 @@ public void testAppendToFileBadArgs() throws Exception {
assertThat(res, not(0));
}
+ @Test (timeout = 30)
+ public void testAppendToFileWithOptionN() throws Exception {
+final int inputFileLength = 1024 * 1024;
+File testRoot = new File(TEST_ROOT_DIR, "testAppendToFileWithOptionN");
+testRoot.mkdirs();
+
+File file1 = new File(testRoot, "file1");
+createLocalFileWithRandomData(inputFileLength, file1);
+
+Configuration conf = new HdfsConfiguration();
+MiniDFSCluster cluster = new
MiniDFSCluster.Builder(conf).numDataNodes(6).build();
+cluster.waitActive();
+
+try {
Review Comment:
Hi @M1eyu2018 , please change the code here and rebase to the latest commit,
as this PR is a bit old.
> Improve appendToFile command: support appending on file with new block
> --
>
> Key: HDFS-16716
> URL: https://issues.apache.org/jira/browse/HDFS-16716
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: guojunhao
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> HDFS client DistributedFileSystem#append supports appending to a file with
> optional create flags.
> However, appendToFile command only supports the default create flag APPEND so
> that append on EC file without NEW_BLOCK create flag is not supported.
> Thus, it's necessary to improve appendToFile command by adding option n for
> it. Option n represents that use NEW_BLOCK create flag while appending file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16716) Improve appendToFile command: support appending on file with new block
[
https://issues.apache.org/jira/browse/HDFS-16716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624112#comment-17624112
]
ASF GitHub Bot commented on HDFS-16716:
---
tomscut commented on code in PR #4697:
URL: https://github.com/apache/hadoop/pull/4697#discussion_r1005123080
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:
##
@@ -3043,6 +3044,101 @@ public void testAppendToFileBadArgs() throws Exception {
assertThat(res, not(0));
}
+ @Test (timeout = 30)
+ public void testAppendToFileWithOptionN() throws Exception {
+final int inputFileLength = 1024 * 1024;
+File testRoot = new File(TEST_ROOT_DIR, "testAppendToFileWithOptionN");
+testRoot.mkdirs();
+
+File file1 = new File(testRoot, "file1");
+createLocalFileWithRandomData(inputFileLength, file1);
+
+Configuration conf = new HdfsConfiguration();
+MiniDFSCluster cluster = new
MiniDFSCluster.Builder(conf).numDataNodes(6).build();
+cluster.waitActive();
+
+try {
Review Comment:
Hi @M1eyu2018 , please change the code here and rebase to the latest commit,
as this PR is a bit old. The other changes look good to me.
> Improve appendToFile command: support appending on file with new block
> --
>
> Key: HDFS-16716
> URL: https://issues.apache.org/jira/browse/HDFS-16716
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: guojunhao
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> HDFS client DistributedFileSystem#append supports appending to a file with
> optional create flags.
> However, appendToFile command only supports the default create flag APPEND so
> that append on EC file without NEW_BLOCK create flag is not supported.
> Thus, it's necessary to improve appendToFile command by adding option n for
> it. Option n represents that use NEW_BLOCK create flag while appending file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16716) Improve appendToFile command: support appending on file with new block
[
https://issues.apache.org/jira/browse/HDFS-16716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624107#comment-17624107
]
ASF GitHub Bot commented on HDFS-16716:
---
tomscut commented on code in PR #4697:
URL: https://github.com/apache/hadoop/pull/4697#discussion_r1005121344
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:
##
@@ -3043,6 +3044,101 @@ public void testAppendToFileBadArgs() throws Exception {
assertThat(res, not(0));
}
+ @Test (timeout = 30)
+ public void testAppendToFileWithOptionN() throws Exception {
+final int inputFileLength = 1024 * 1024;
+File testRoot = new File(TEST_ROOT_DIR, "testAppendToFileWithOptionN");
+testRoot.mkdirs();
+
+File file1 = new File(testRoot, "file1");
+createLocalFileWithRandomData(inputFileLength, file1);
+
+Configuration conf = new HdfsConfiguration();
+MiniDFSCluster cluster = new
MiniDFSCluster.Builder(conf).numDataNodes(6).build();
+cluster.waitActive();
+
+try {
Review Comment:
```suggestion
try (MiniDFSCluster cluster = new
MiniDFSCluster.Builder(conf).numDataNodes(6).build()){
```
> Improve appendToFile command: support appending on file with new block
> --
>
> Key: HDFS-16716
> URL: https://issues.apache.org/jira/browse/HDFS-16716
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: guojunhao
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> HDFS client DistributedFileSystem#append supports appending to a file with
> optional create flags.
> However, appendToFile command only supports the default create flag APPEND so
> that append on EC file without NEW_BLOCK create flag is not supported.
> Thus, it's necessary to improve appendToFile command by adding option n for
> it. Option n represents that use NEW_BLOCK create flag while appending file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16819) Remove the redundant write lock in FsDatasetImpl#createTemporary
[
https://issues.apache.org/jira/browse/HDFS-16819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624104#comment-17624104
]
ASF GitHub Bot commented on HDFS-16819:
---
tomscut commented on code in PR #5074:
URL: https://github.com/apache/hadoop/pull/5074#discussion_r1005118073
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:
##
@@ -1912,9 +1908,8 @@ public ReplicaHandler createTemporary(StorageType
storageType,
return new ReplicaHandler(newReplicaInfo, ref);
} finally {
if (dataNodeMetrics != null) {
-// Create temporary operation hold write lock twice.
-long createTemporaryOpMs = Time.monotonicNow() - startHoldLockTimeMs
-+ holdLockTimeMs;
+// Create temporary operation hold write lock once.
+long createTemporaryOpMs = Time.monotonicNow() - startHoldLockTimeMs;
Review Comment:
I think `createTemporaryOpMs` should be `Time.monotonicNow() - startTimeMs`;
> Remove the redundant write lock in FsDatasetImpl#createTemporary
> --
>
> Key: HDFS-16819
> URL: https://issues.apache.org/jira/browse/HDFS-16819
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> In FsDatasetImpl#createTemporary Line_1840 the writeLock here seems useless.
> The readLock is already held in volumeMap.get(). From the code logic point
> of view, the writeLock here maybe to remove
> {code:java}
> public ReplicaHandler createTemporary(StorageType storageType,
> String storageId, ExtendedBlock b, boolean isTransfer)
> throws IOException {
> long startTimeMs = Time.monotonicNow();
> long writerStopTimeoutMs = datanode.getDnConf().getXceiverStopTimeout();
> ReplicaInfo lastFoundReplicaInfo = null;
> boolean isInPipeline = false;
> do {
>try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.BLOCK_POOl,
>b.getBlockPoolId())) { //the writeLock here maybe to remove
> ReplicaInfo currentReplicaInfo =
> volumeMap.get(b.getBlockPoolId(), b.getBlockId());
> if (currentReplicaInfo == lastFoundReplicaInfo) {
>break;
> } else {
>isInPipeline = currentReplicaInfo.getState() == ReplicaState.TEMPORARY
>|| currentReplicaInfo.getState() == ReplicaState.RBW;
>/*
> * If the current block is not PROVIDED and old, reject.
> * else If transfer request, then accept it.
> * else if state is not RBW/Temporary, then reject
> * If current block is PROVIDED, ignore the replica.
> */
>if (((currentReplicaInfo.getGenerationStamp() >= b
>.getGenerationStamp()) || (!isTransfer && !isInPipeline))
>&& !isReplicaProvided(currentReplicaInfo)) {
> throw new ReplicaAlreadyExistsException("Block " + b
> + " already exists in state " + currentReplicaInfo.getState()
> + " and thus cannot be created.");
>}
>lastFoundReplicaInfo = currentReplicaInfo;
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16811) Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
[
https://issues.apache.org/jira/browse/HDFS-16811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624096#comment-17624096
]
ASF GitHub Bot commented on HDFS-16811:
---
tomscut commented on code in PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#discussion_r1005110587
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##
@@ -2321,6 +2324,9 @@ protected String reconfigurePropertyImpl(String property,
String newVal)
return reconfigureSlowNodesParameters(datanodeManager, property, newVal);
} else if (property.equals(DFS_BLOCK_INVALIDATE_LIMIT_KEY)) {
return reconfigureBlockInvalidateLimit(datanodeManager, property,
newVal);
+} else if
(property.equals(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT)) {
Review Comment:
> Hi @tomscut, thanks for your comment. yeah, can create new method
reconfigureDecommissionBackoffMonitorParameters() to update
DecommissionBackoffMonitor related parameters.
>
> [HDFS-16810](https://issues.apache.org/jira/browse/HDFS-16810) to support
‘DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK_DEFAULT’
reconfigurable
>
> Do we need to merge 2 issues into one to support?
I think it could be merged together to make it clearer. Thanks.
> Support to make dfs.namenode.decommission.backoff.monitor.pending.limit
> reconfigurable
> ---
>
> Key: HDFS-16811
> URL: https://issues.apache.org/jira/browse/HDFS-16811
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> When the Backoff monitor is enabled, the parameter
> dfs.namenode.decommission.backoff.monitor.pending.limit can be dynamically
> adjusted to determines the maximum number of blocks related to decommission
> and maintenance operations that can be loaded into the replication queue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16816) RBF: auto-create user home dir for trash paths by router
[
https://issues.apache.org/jira/browse/HDFS-16816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624090#comment-17624090
]
ASF GitHub Bot commented on HDFS-16816:
---
hadoop-yetus commented on PR #5071:
URL: https://github.com/apache/hadoop/pull/5071#issuecomment-1291288174
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 35s | | trunk passed |
| +1 :green_heart: | compile | 0m 51s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 38s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5071/4/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 4
unchanged - 0 fixed = 5 total (was 4) |
| +1 :green_heart: | mvnsite | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 2s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 35m 43s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 142m 34s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5071/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5071 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux c063aa131c1b 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8563b455f6c32f53fa683f43638fdda20e169846 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5071/4/testReport/ |
| Max. process+thread count | 3174 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space
[
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624087#comment-17624087
]
ASF GitHub Bot commented on HDFS-3570:
--
ashutoshcipher commented on PR #5044:
URL: https://github.com/apache/hadoop/pull/5044#issuecomment-1291278089
@aajisaka @ZanderXu, Please help in reviewing this PR in your free time.
Thanks.
> Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used
> space
>
>
> Key: HDFS-3570
> URL: https://issues.apache.org/jira/browse/HDFS-3570
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer mover
>Affects Versions: 2.0.0-alpha
>Reporter: Harsh J
>Assignee: Ashutosh Gupta
>Priority: Minor
> Labels: pull-request-available
> Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch,
> HDFS-3570.aash.1.patch
>
>
> Report from a user here:
> https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
> post archived at http://pastebin.com/eVFkk0A0
> This user had a specific DN that had a large non-DFS usage among
> dfs.data.dirs, and very little DFS usage (which is computed against total
> possible capacity).
> Balancer apparently only looks at the usage, and ignores to consider that
> non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a
> DFS Usage report from DN is 8% only, its got a lot of free space to write
> more blocks, when that isn't true as shown by the case of this user. It went
> on scheduling writes to the DN to balance it out, but the DN simply can't
> accept any more blocks as a result of its disks' state.
> I think it would be better if we _computed_ the actual utilization based on
> {{(100-(actual remaining space))/(capacity)}}, as opposed to the current
> {{(dfs used)/(capacity)}}. Thoughts?
> This isn't very critical, however, cause it is very rare to see DN space
> being used for non DN data, but it does expose a valid bug.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16818) RBF TestRouterRPCMultipleDestinationMountTableResolver non-deterministic unit tests failures
[
https://issues.apache.org/jira/browse/HDFS-16818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624081#comment-17624081
]
Xing Lin commented on HDFS-16818:
-
This unit test failure is non-deterministic. May be related with HDFS-16740.
> RBF TestRouterRPCMultipleDestinationMountTableResolver non-deterministic unit
> tests failures
>
>
> Key: HDFS-16818
> URL: https://issues.apache.org/jira/browse/HDFS-16818
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Affects Versions: 3.4.0
>Reporter: Xing Lin
>Priority: Major
>
> TestRouterRPCMultipleDestinationMountTableResolver fails a couple of times
> nondeterministically when run multiple times.
> I repeated the following commands for 10+ times against
> 454157a3844cdd6c92ef650af6c3b323cbec88af in trunk and observed two types of
> failed runs.
> {code:java}
> mvn test -Dtest="TestRouterRPCMultipleDestinationMountTableResolver"{code}
>
> Failed run 1 output:
> {code:java}
> [ERROR] Failures:
> [ERROR]
> TestRouterRPCMultipleDestinationMountTableResolver.testInvocationHashAllOrder:177->testInvocation:221->testDirec
> toryAndFileLevelInvocation:296->verifyDirectoryLevelInvocations:395
> expected:<[COLD]> but was:<[HOT]>
> [ERROR]
> TestRouterRPCMultipleDestinationMountTableResolver.testInvocationHashOrder:193->testInvocation:221->testDirector
> yAndFileLevelInvocation:298->verifyDirectoryLevelInvocations:395
> expected:<[COLD]> but was:<[HOT]>
> [ERROR]
> TestRouterRPCMultipleDestinationMountTableResolver.testInvocationLocalOrder:201->testInvocation:221->testDirecto
> ryAndFileLevelInvocation:296->verifyDirectoryLevelInvocations:395
> expected:<[COLD]> but was:<[HOT]>
> [ERROR]
> TestRouterRPCMultipleDestinationMountTableResolver.testInvocationRandomOrder:185->testInvocation:221->testDirect
> oryAndFileLevelInvocation:296->verifyDirectoryLevelInvocations:395
> expected:<[COLD]> but was:<[HOT]>
> [ERROR]
> TestRouterRPCMultipleDestinationMountTableResolver.testInvocationSpaceOrder:169->testInvocation:221->testDirecto
> ryAndFileLevelInvocation:296->verifyDirectoryLevelInvocations:395
> expected:<[COLD]> but was:<[HOT]>
> [INFO]
> [ERROR] Tests run: 18, Failures: 5, Errors: 0, Skipped: 0{code}
>
> Failed run 2 output:
> {code:java}
> [ERROR] Failures:
> [ERROR]
> TestRouterRPCMultipleDestinationMountTableResolver.testECMultipleDestinations:430
> [ERROR] Errors:
> [ERROR]
> TestRouterRPCMultipleDestinationMountTableResolver.testInvocationHashAllOrder:177->testInvocation:221->testDirec
> toryAndFileLevelInvocation:296->verifyDirectoryLevelInvocations:397
> NullPointer
> [ERROR]
> TestRouterRPCMultipleDestinationMountTableResolver.testInvocationHashOrder:193->testInvocation:221->testDirector
> yAndFileLevelInvocation:298->verifyDirectoryLevelInvocations:397 NullPointer
> [ERROR]
> TestRouterRPCMultipleDestinationMountTableResolver.testInvocationLocalOrder:201->testInvocation:221->testDirecto
> ryAndFileLevelInvocation:296->verifyDirectoryLevelInvocations:397 NullPointer
> [ERROR]
> TestRouterRPCMultipleDestinationMountTableResolver.testInvocationRandomOrder:185->testInvocation:221->testDirect
> oryAndFileLevelInvocation:296->verifyDirectoryLevelInvocations:397 NullPointer
> [ERROR]
> TestRouterRPCMultipleDestinationMountTableResolver.testInvocationSpaceOrder:169->testInvocation:221->testDirecto
> ryAndFileLevelInvocation:296->verifyDirectoryLevelInvocations:397 NullPointer
> [INFO]
> [ERROR] Tests run: 18, Failures: 1, Errors: 5, Skipped: 0{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16740) Mini cluster test flakiness
[
https://issues.apache.org/jira/browse/HDFS-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624080#comment-17624080
]
ASF GitHub Bot commented on HDFS-16740:
---
xinglin commented on code in PR #4835:
URL: https://github.com/apache/hadoop/pull/4835#discussion_r1005061516
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommissionWithStriped.java:
##
@@ -117,10 +121,8 @@ protected Configuration createConfiguration() {
public void setup() throws IOException {
conf = createConfiguration();
// Set up the hosts/exclude files.
-localFileSys = FileSystem.getLocal(conf);
-Path workingDir = localFileSys.getWorkingDirectory();
-decommissionDir = new Path(workingDir,
-PathUtils.getTestDirName(getClass()) + "/work-dir/decommission");
+localFileSys = FileSystem.get(baseDir.getRoot().toURI(), conf);
Review Comment:
Will FileSystem.get() return a LocalFileSystem object here? what scheme does
baseDir.getRoot().toURI() return?
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java:
##
@@ -83,37 +83,18 @@ public static void runCmd(DFSAdmin dfsadmin, boolean
success,
}
@Rule
- public TemporaryFolder folder = new TemporaryFolder();
-
- /**
- * Create a default HDFS configuration which has test-specific data
directories. This is
- * intended to protect against interactions between test runs that might
corrupt results. Each
- * test run's data is automatically cleaned-up by JUnit.
- *
- * @return a default configuration with test-specific data directories
- */
- public Configuration getHdfsConfiguration() throws IOException {
-Configuration conf = new HdfsConfiguration();
-
-// Override the file system locations with test-specific temporary folders
-conf.set(DFSConfigKeys.DFS_NAMENODE_NAME_DIR_KEY,
Review Comment:
Why is the case that we no longer need to set these properties anymore? Is
it because now NAME_DIR/DATA_DIR/CHECKPOINT_DIR will all defaulted to be within
the new baseDir?
> Mini cluster test flakiness
> ---
>
> Key: HDFS-16740
> URL: https://issues.apache.org/jira/browse/HDFS-16740
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, test
>Affects Versions: 3.4.0, 3.3.5
>Reporter: Steve Vaughan
>Assignee: Steve Vaughan
>Priority: Major
> Labels: pull-request-available
>
> Mini clusters used during HDFS unit tests are reporting test failures that do
> not appear to be directly related to submitted changes. The failures are the
> result of either interactions between tests run in parallel, or tests which
> share common disk space for tests. In all cases, the tests can be run
> individually serially without any errors. Addressing this issue will
> simplify future submissions by eliminating the confusion introduced by these
> unrelated test failures.
> We can apply lessons recently from TestRollingUpgrade, which was recently
> patched to unblock a recent submission. The fixes involved changing the HDFS
> configuration to use temporary disk space for each individual tests, and
> using try-with-resources to ensure that clusters were shutdown cleanly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16815) Error occurred in processing CacheManagerSection for xml parsing fsimage
[
https://issues.apache.org/jira/browse/HDFS-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624068#comment-17624068
]
ASF GitHub Bot commented on HDFS-16815:
---
jojochuang commented on code in PR #5069:
URL: https://github.com/apache/hadoop/pull/5069#discussion_r1005012357
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java:
##
@@ -1410,4 +1410,49 @@ public void testOfflineImageViewerForECPolicies() throws
Exception {
}
}
}
+
+ /**
+ * Tests that ReverseXML processor doesn't accept XML files with the
CacheManagerSection contains pool or directive tag
+ */
+ @Test
+ public void testReverseXmlWithCacheManagerSection() throws Throwable {
+File imageXml = new File(tempDir, "imageXml.xml");
+PrintWriter writer = new PrintWriter(imageXml, "UTF-8");
+try {
+ writer.println("");
+ writer.println("");
+ writer.println("");
+ writer.println(String.format("%d",
+ NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION));
+ writer.println("1");
+ writer.println("" +
+ "545bbef596c06af1c3c8dca1ce29096a64608478");
+ writer.println("");
+
+ writer.println("");
+ writer.println("1");
+ writer.println("0");
+ writer.println("1");
+
+ writer.println("");
+ writer.println("0");
+ writer.println("1");
+ writer.println("2");
+ writer.println("3");
+ writer.println("4");
+ writer.println("45554");
+ writer.println("");
+ writer.println("");
+ writer.println("");
+} finally {
+ writer.close();
+}
+try {
+ OfflineImageReconstructor.run(imageXml.getAbsolutePath(),
+ imageXml.getAbsolutePath() + ".out");
+ Assert.fail("missing some tags in pool filed");
+} catch (Throwable t) {
+ GenericTestUtils.assertExceptionContains("missing some tags in pool
filed", t);
+}
Review Comment:
This assertion logic looks super convoluted.
```suggestion
OfflineImageReconstructor.run(imageXml.getAbsolutePath(),
imageXml.getAbsolutePath() + ".out");
```
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java:
##
@@ -1410,4 +1410,49 @@ public void testOfflineImageViewerForECPolicies() throws
Exception {
}
}
}
+
+ /**
+ * Tests that ReverseXML processor doesn't accept XML files with the
CacheManagerSection contains pool or directive tag
+ */
+ @Test
+ public void testReverseXmlWithCacheManagerSection() throws Throwable {
+File imageXml = new File(tempDir, "imageXml.xml");
+PrintWriter writer = new PrintWriter(imageXml, "UTF-8");
+try {
Review Comment:
```suggestion
try (PrintWriter writer = new PrintWriter(imageXml, "UTF-8")) {
```
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java:
##
@@ -1410,4 +1410,49 @@ public void testOfflineImageViewerForECPolicies() throws
Exception {
}
}
}
+
+ /**
+ * Tests that ReverseXML processor doesn't accept XML files with the
CacheManagerSection contains pool or directive tag
+ */
+ @Test
+ public void testReverseXmlWithCacheManagerSection() throws Throwable {
+File imageXml = new File(tempDir, "imageXml.xml");
+PrintWriter writer = new PrintWriter(imageXml, "UTF-8");
+try {
+ writer.println("");
+ writer.println("");
+ writer.println("");
+ writer.println(String.format("%d",
+ NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION));
+ writer.println("1");
+ writer.println("" +
+ "545bbef596c06af1c3c8dca1ce29096a64608478");
+ writer.println("");
+
+ writer.println("");
+ writer.println("1");
+ writer.println("0");
+ writer.println("1");
+
+ writer.println("");
+ writer.println("0");
+ writer.println("1");
+ writer.println("2");
+ writer.println("3");
+ writer.println("4");
+ writer.println("45554");
+ writer.println("");
+ writer.println("");
+ writer.println("");
+} finally {
+ writer.close();
+}
Review Comment:
```suggestion
}
```
> Error occurred in processing CacheManagerSection for xml parsing fsimage
>
>
> Key: HDFS-16815
> URL: https://issues.apache.org/jira/browse/HDFS-16815
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: MeiJing
>Priority: Minor
> Labels: pull-request-available
> Attachments: image-2022-10-24-10-34-07-288.png,
> image-2022-10-24-10-43-57-652.png, image-2022-10-24-10-44-51-234.png
>
>
> For
[jira] [Commented] (HDFS-16802) Print options when accessing ClientProtocol#rename2()
[
https://issues.apache.org/jira/browse/HDFS-16802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624065#comment-17624065
]
ASF GitHub Bot commented on HDFS-16802:
---
jojochuang commented on code in PR #5013:
URL: https://github.com/apache/hadoop/pull/5013#discussion_r1005007173
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java:
##
@@ -247,10 +247,8 @@ static RenameResult renameToInt(
throws IOException {
String src = srcArg;
String dst = dstArg;
-if (NameNode.stateChangeLog.isDebugEnabled()) {
- NameNode.stateChangeLog.debug("DIR* NameSystem.renameTo: with options -"
+
- " " + src + " to " + dst);
-}
+NameNode.stateChangeLog.debug("DIR* NameSystem.renameTo: with options={}
{} to {}",
+Arrays.toString(options), src, dst);
Review Comment:
not sure about this...
Wouldn't it incur overhead because it has to compose a string before
determining to log or not?
> Print options when accessing ClientProtocol#rename2()
> -
>
> Key: HDFS-16802
> URL: https://issues.apache.org/jira/browse/HDFS-16802
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.3.4
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
> Labels: pull-request-available
>
> When accessing ClientProtocol#rename2(), the carried options cannot be seen
> in the log. Here is some log information:
> {code:java}
> 2022-10-13 10:21:10,727 [Listener at localhost/59732] DEBUG hdfs.StateChange
> (FSDirRenameOp.java:renameToInt(255)) - DIR* NameSystem.renameTo: with
> options - /testNamenodeRetryCache/testRename2/src to
> /testNamenodeRetryCache/testRename2/target
> {code}
> We should improve this, maybe printing options would be better.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16809) EC striped block is not sufficient when doing in maintenance
[ https://issues.apache.org/jira/browse/HDFS-16809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-16809: --- Component/s: erasure-coding (was: hdfs) > EC striped block is not sufficient when doing in maintenance > > > Key: HDFS-16809 > URL: https://issues.apache.org/jira/browse/HDFS-16809 > Project: Hadoop HDFS > Issue Type: Bug > Components: ec, erasure-coding >Reporter: dingshun >Assignee: dingshun >Priority: Major > Labels: pull-request-available > > When doing maintenance, ec striped block is not sufficient, which will lead > to miss block -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16809) EC striped block is not sufficient when doing in maintenance
[ https://issues.apache.org/jira/browse/HDFS-16809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reassigned HDFS-16809: -- Assignee: dingshun > EC striped block is not sufficient when doing in maintenance > > > Key: HDFS-16809 > URL: https://issues.apache.org/jira/browse/HDFS-16809 > Project: Hadoop HDFS > Issue Type: Bug > Components: ec, hdfs >Reporter: dingshun >Assignee: dingshun >Priority: Major > Labels: pull-request-available > > When doing maintenance, ec striped block is not sufficient, which will lead > to miss block -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16816) RBF: auto-create user home dir for trash paths by router
[ https://issues.apache.org/jira/browse/HDFS-16816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623955#comment-17623955 ] ASF GitHub Bot commented on HDFS-16816: --- hadoop-yetus commented on PR #5071: URL: https://github.com/apache/hadoop/pull/5071#issuecomment-1290933389 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | docker | 4m 23s | | Docker failed to build run-specific yetus/hadoop:tp-29905}. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/5071 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5071/3/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > RBF: auto-create user home dir for trash paths by router > > > Key: HDFS-16816 > URL: https://issues.apache.org/jira/browse/HDFS-16816 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Xing Lin >Priority: Minor > Labels: pull-request-available > > In RBF, trash files are moved to trash root under user's home dir at the > corresponding namespace/namenode where the files reside. This was added in > HDFS-16024. When the user home dir is not created before-hand at a namenode, > we run into permission denied exceptions when trying to create the parent dir > for the trash file before moving the file into it. We propose to enhance > Router, to auto-create a user home's dir at the namenode for trash paths, > using router's identity (which is assumed to be a super-user). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-8510) Provide different timeout settings for hdfs dfsadmin -getDatanodeInfo.
[
https://issues.apache.org/jira/browse/HDFS-8510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-8510.
-
Resolution: Won't Fix
This is an old improvement proposal that I'm no longer planning on
implementing. I'm going to close the issue. If anyone else would find it
useful, please feel free to reopen and reassign. I'd be happy to help with code
review.
> Provide different timeout settings for hdfs dfsadmin -getDatanodeInfo.
> --
>
> Key: HDFS-8510
> URL: https://issues.apache.org/jira/browse/HDFS-8510
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
>
> During a rolling upgrade, an administrator runs {{hdfs dfsadmin
> -getDatanodeInfo}} to check if a DataNode has stopped. Currently, this
> operation is subject to the RPC connection retries defined in
> {{ipc.client.connect.max.retries}} and {{ipc.client.connect.retry.interval}}.
> This issue proposes adding separate configuration properties to control the
> retries for this operation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-4289) FsDatasetImpl#updateReplicaUnderRecovery throws errors validating replica byte count on Windows
[
https://issues.apache.org/jira/browse/HDFS-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-4289.
-
Resolution: Won't Fix
I'm no longer actively working on this. I no longer have easy access to a
Windows environment to make Windows-specific changes, or even to confirm that
this test failure still happens. It's a very old issue with no recent activity,
so I'm going to assume it's no longer relevant and close it out. If it's still
an ongoing issue that a Windows developer wants to pick up, please feel free to
reopen and reassign.
> FsDatasetImpl#updateReplicaUnderRecovery throws errors validating replica
> byte count on Windows
> ---
>
> Key: HDFS-4289
> URL: https://issues.apache.org/jira/browse/HDFS-4289
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: trunk-win
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
>
> {{FsDatasetImpl#updateReplicaUnderRecovery}} throws errors validating replica
> byte count on Windows. This can be seen by running
> {{TestBalancerWithNodeGroup#testBalancerWithRackLocality}}, which fails on
> Windows.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-3296) Running libhdfs tests in mac fails
[
https://issues.apache.org/jira/browse/HDFS-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth reassigned HDFS-3296:
---
Assignee: (was: Chris Nauroth)
I'm going to unassign this, because I'm no longer actively working on it. I see
a new patch revision came in from [~jzhuge] a while ago. John (or others),
please feel free to take it if you're working on it.
> Running libhdfs tests in mac fails
> --
>
> Key: HDFS-3296
> URL: https://issues.apache.org/jira/browse/HDFS-3296
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Reporter: Amareshwari Sriramadasu
>Priority: Major
> Attachments: HDFS-3296.001.patch, HDFS-3296.002.patch,
> HDFS-3296.003.patch, HDFS-3296.004.patch
>
>
> Running "ant -Dcompile.c++=true -Dlibhdfs=true test-c++-libhdfs" on Mac fails
> with following error:
> {noformat}
> [exec] dyld: lazy symbol binding failed: Symbol not found:
> _JNI_GetCreatedJavaVMs
> [exec] Referenced from:
> /Users/amareshwari.sr/workspace/hadoop/build/c++/Mac_OS_X-x86_64-64/lib/libhdfs.0.dylib
> [exec] Expected in: flat namespace
> [exec]
> [exec] dyld: Symbol not found: _JNI_GetCreatedJavaVMs
> [exec] Referenced from:
> /Users/amareshwari.sr/workspace/hadoop/build/c++/Mac_OS_X-x86_64-64/lib/libhdfs.0.dylib
> [exec] Expected in: flat namespace
> [exec]
> [exec]
> /Users/amareshwari.sr/workspace/hadoop/src/c++/libhdfs/tests/test-libhdfs.sh:
> line 122: 39485 Trace/BPT trap: 5 CLASSPATH=$HADOOP_CONF_DIR:$CLASSPATH
> LD_PRELOAD="$LIB_JVM_DIR/libjvm.so:$LIBHDFS_INSTALL_DIR/libhdfs.so:"
> $LIBHDFS_BUILD_DIR/$HDFS_TEST
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16740) Mini cluster test flakiness
[
https://issues.apache.org/jira/browse/HDFS-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623838#comment-17623838
]
ASF GitHub Bot commented on HDFS-16740:
---
snmvaughan commented on code in PR #4835:
URL: https://github.com/apache/hadoop/pull/4835#discussion_r1004514283
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/MiniJournalCluster.java:
##
@@ -64,7 +65,17 @@ public static class Builder {
public Builder(Configuration conf) {
this.conf = conf;
}
-
+
+public Builder(Configuration conf, TemporaryFolder baseDir) {
Review Comment:
I've stripped all the JUnit dependencies, including one instance that used a
JUnit assertion.
> Mini cluster test flakiness
> ---
>
> Key: HDFS-16740
> URL: https://issues.apache.org/jira/browse/HDFS-16740
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, test
>Affects Versions: 3.4.0, 3.3.5
>Reporter: Steve Vaughan
>Assignee: Steve Vaughan
>Priority: Major
> Labels: pull-request-available
>
> Mini clusters used during HDFS unit tests are reporting test failures that do
> not appear to be directly related to submitted changes. The failures are the
> result of either interactions between tests run in parallel, or tests which
> share common disk space for tests. In all cases, the tests can be run
> individually serially without any errors. Addressing this issue will
> simplify future submissions by eliminating the confusion introduced by these
> unrelated test failures.
> We can apply lessons recently from TestRollingUpgrade, which was recently
> patched to unblock a recent submission. The fixes involved changing the HDFS
> configuration to use temporary disk space for each individual tests, and
> using try-with-resources to ensure that clusters were shutdown cleanly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16740) Mini cluster test flakiness
[
https://issues.apache.org/jira/browse/HDFS-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623836#comment-17623836
]
ASF GitHub Bot commented on HDFS-16740:
---
steveloughran commented on code in PR #4835:
URL: https://github.com/apache/hadoop/pull/4835#discussion_r1004510981
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/MiniJournalCluster.java:
##
@@ -64,7 +65,17 @@ public static class Builder {
public Builder(Configuration conf) {
this.conf = conf;
}
-
+
+public Builder(Configuration conf, TemporaryFolder baseDir) {
Review Comment:
no idea where it is used, but in tests in other applications, they may not
use junit or may be on a different version of it
> Mini cluster test flakiness
> ---
>
> Key: HDFS-16740
> URL: https://issues.apache.org/jira/browse/HDFS-16740
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, test
>Affects Versions: 3.4.0, 3.3.5
>Reporter: Steve Vaughan
>Assignee: Steve Vaughan
>Priority: Major
> Labels: pull-request-available
>
> Mini clusters used during HDFS unit tests are reporting test failures that do
> not appear to be directly related to submitted changes. The failures are the
> result of either interactions between tests run in parallel, or tests which
> share common disk space for tests. In all cases, the tests can be run
> individually serially without any errors. Addressing this issue will
> simplify future submissions by eliminating the confusion introduced by these
> unrelated test failures.
> We can apply lessons recently from TestRollingUpgrade, which was recently
> patched to unblock a recent submission. The fixes involved changing the HDFS
> configuration to use temporary disk space for each individual tests, and
> using try-with-resources to ensure that clusters were shutdown cleanly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16819) Remove the redundant write lock in FsDatasetImpl#createTemporary
[
https://issues.apache.org/jira/browse/HDFS-16819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623776#comment-17623776
]
ASF GitHub Bot commented on HDFS-16819:
---
hadoop-yetus commented on PR #5074:
URL: https://github.com/apache/hadoop/pull/5074#issuecomment-1290414892
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 54s | | trunk passed |
| +1 :green_heart: | compile | 1m 38s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 32s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 19s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 58s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 39s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 246m 10s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5074/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 12s | | The patch does not
generate ASF License warnings. |
| | | 370m 15s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5074/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5074 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6b1ee7d6161a 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 09dacd784a24311952ddbb90b8b76af17f95f5b1 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5074/1/testReport/ |
| Max. process+thread count | 2966 (vs. ulimit of 5500)
[jira] [Commented] (HDFS-16813) Remove parameter validation logic such as dfs.namenode.decommission.blocks.per.interval in DatanodeAdminManager#activate
[
https://issues.apache.org/jira/browse/HDFS-16813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623726#comment-17623726
]
ASF GitHub Bot commented on HDFS-16813:
---
haiyang1987 commented on code in PR #5063:
URL: https://github.com/apache/hadoop/pull/5063#discussion_r1004270545
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java:
##
@@ -113,6 +113,17 @@ protected void processConf() {
numBlocksPerCheck =
DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_DEFAULT;
}
+
+final String deprecatedKey =
+"dfs.namenode.decommission.nodes.per.interval";
+final String strNodes = conf.get(deprecatedKey);
+if (strNodes != null) {
+ LOG.warn("Deprecated configuration key {} will be ignored.",
Review Comment:
ok. I will update later.
> Remove parameter validation logic such as
> dfs.namenode.decommission.blocks.per.interval in DatanodeAdminManager#activate
>
>
> Key: HDFS-16813
> URL: https://issues.apache.org/jira/browse/HDFS-16813
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> In DatanodeAdminManager#activate
> {code:java}
> int blocksPerInterval = conf.getInt(
> DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_KEY,
> DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_DEFAULT);
> final String deprecatedKey =
> "dfs.namenode.decommission.nodes.per.interval";
> final String strNodes = conf.get(deprecatedKey);
> if (strNodes != null) {
> LOG.warn("Deprecated configuration key {} will be ignored.",
> deprecatedKey);
> LOG.warn("Please update your configuration to use {} instead.",
> DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_KEY);
> }
> checkArgument(blocksPerInterval > 0,
> "Must set a positive value for "
> + DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_KEY);
> final int maxConcurrentTrackedNodes = conf.getInt(
> DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_MAX_CONCURRENT_TRACKED_NODES,
> DFSConfigKeys
> .DFS_NAMENODE_DECOMMISSION_MAX_CONCURRENT_TRACKED_NODES_DEFAULT);
> checkArgument(maxConcurrentTrackedNodes >= 0, "Cannot set a negative " +
> "value for "
> + DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_MAX_CONCURRENT_TRACKED_NODES);
> {code}
> there is no need for parameters
> dfs.namenode.decommission.blocks.per.interval and
> dfs.namenode.decommission.max.concurrent.tracked.nodes to verify.
> Because the parameters are processed in DatanodeAdminMonitorBase and
> DatanodeAdminDefaultMonitor.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16813) Remove parameter validation logic such as dfs.namenode.decommission.blocks.per.interval in DatanodeAdminManager#activate
[
https://issues.apache.org/jira/browse/HDFS-16813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623725#comment-17623725
]
ASF GitHub Bot commented on HDFS-16813:
---
haiyang1987 commented on code in PR #5063:
URL: https://github.com/apache/hadoop/pull/5063#discussion_r1004269693
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java:
##
@@ -113,6 +113,17 @@ protected void processConf() {
numBlocksPerCheck =
DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_DEFAULT;
}
+
+final String deprecatedKey =
+"dfs.namenode.decommission.nodes.per.interval";
Review Comment:
sorry sir, I didn't understand you here.
> Remove parameter validation logic such as
> dfs.namenode.decommission.blocks.per.interval in DatanodeAdminManager#activate
>
>
> Key: HDFS-16813
> URL: https://issues.apache.org/jira/browse/HDFS-16813
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> In DatanodeAdminManager#activate
> {code:java}
> int blocksPerInterval = conf.getInt(
> DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_KEY,
> DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_DEFAULT);
> final String deprecatedKey =
> "dfs.namenode.decommission.nodes.per.interval";
> final String strNodes = conf.get(deprecatedKey);
> if (strNodes != null) {
> LOG.warn("Deprecated configuration key {} will be ignored.",
> deprecatedKey);
> LOG.warn("Please update your configuration to use {} instead.",
> DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_KEY);
> }
> checkArgument(blocksPerInterval > 0,
> "Must set a positive value for "
> + DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_KEY);
> final int maxConcurrentTrackedNodes = conf.getInt(
> DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_MAX_CONCURRENT_TRACKED_NODES,
> DFSConfigKeys
> .DFS_NAMENODE_DECOMMISSION_MAX_CONCURRENT_TRACKED_NODES_DEFAULT);
> checkArgument(maxConcurrentTrackedNodes >= 0, "Cannot set a negative " +
> "value for "
> + DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_MAX_CONCURRENT_TRACKED_NODES);
> {code}
> there is no need for parameters
> dfs.namenode.decommission.blocks.per.interval and
> dfs.namenode.decommission.max.concurrent.tracked.nodes to verify.
> Because the parameters are processed in DatanodeAdminMonitorBase and
> DatanodeAdminDefaultMonitor.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16813) Remove parameter validation logic such as dfs.namenode.decommission.blocks.per.interval in DatanodeAdminManager#activate
[
https://issues.apache.org/jira/browse/HDFS-16813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623724#comment-17623724
]
ASF GitHub Bot commented on HDFS-16813:
---
haiyang1987 commented on code in PR #5063:
URL: https://github.com/apache/hadoop/pull/5063#discussion_r1004267679
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java:
##
@@ -113,6 +113,17 @@ protected void processConf() {
numBlocksPerCheck =
DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_DEFAULT;
}
+
+final String deprecatedKey =
Review Comment:
Hi @goiri thanks for your review.
currently already removed "dfs.namenode.decommission.nodes.per.interval"
in DFSConfigKeys ,
so no need to mark it, right?
> Remove parameter validation logic such as
> dfs.namenode.decommission.blocks.per.interval in DatanodeAdminManager#activate
>
>
> Key: HDFS-16813
> URL: https://issues.apache.org/jira/browse/HDFS-16813
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> In DatanodeAdminManager#activate
> {code:java}
> int blocksPerInterval = conf.getInt(
> DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_KEY,
> DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_DEFAULT);
> final String deprecatedKey =
> "dfs.namenode.decommission.nodes.per.interval";
> final String strNodes = conf.get(deprecatedKey);
> if (strNodes != null) {
> LOG.warn("Deprecated configuration key {} will be ignored.",
> deprecatedKey);
> LOG.warn("Please update your configuration to use {} instead.",
> DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_KEY);
> }
> checkArgument(blocksPerInterval > 0,
> "Must set a positive value for "
> + DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_KEY);
> final int maxConcurrentTrackedNodes = conf.getInt(
> DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_MAX_CONCURRENT_TRACKED_NODES,
> DFSConfigKeys
> .DFS_NAMENODE_DECOMMISSION_MAX_CONCURRENT_TRACKED_NODES_DEFAULT);
> checkArgument(maxConcurrentTrackedNodes >= 0, "Cannot set a negative " +
> "value for "
> + DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_MAX_CONCURRENT_TRACKED_NODES);
> {code}
> there is no need for parameters
> dfs.namenode.decommission.blocks.per.interval and
> dfs.namenode.decommission.max.concurrent.tracked.nodes to verify.
> Because the parameters are processed in DatanodeAdminMonitorBase and
> DatanodeAdminDefaultMonitor.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16716) Improve appendToFile command: support appending on file with new block
[
https://issues.apache.org/jira/browse/HDFS-16716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623709#comment-17623709
]
ASF GitHub Bot commented on HDFS-16716:
---
tomscut commented on code in PR #4697:
URL: https://github.com/apache/hadoop/pull/4697#discussion_r1003970562
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java:
##
@@ -1544,6 +1544,39 @@ public FSDataOutputStream append(Path f, int bufferSize)
throws IOException {
public abstract FSDataOutputStream append(Path f, int bufferSize,
Progressable progress) throws IOException;
+ /**
+ * Append to an existing file (optional operation).
+ * @param f the existing file to be appended.
+ * @param appendToNewBlock whether to append data to a new block
+ * instead of the end of the last partial block
+ * @throws IOException IO failure
+ * @throws UnsupportedOperationException if the operation is unsupported
+ * (default).
+ * @return output stream.
+ */
+ public FSDataOutputStream append(Path f, boolean appendToNewBlock) throws
IOException {
+return append(f, getConf().getInt(IO_FILE_BUFFER_SIZE_KEY,
+IO_FILE_BUFFER_SIZE_DEFAULT), null, appendToNewBlock);
+ }
+
+ /**
+ * Append to an existing file (optional operation).
+ * This function is used for being overridden by some FileSystem like
DistributedFileSystem
+ * @param f the existing file to be appended.
+ * @param bufferSize the size of the buffer to be used.
+ * @param progress for reporting progress if it is not null.
+ * @param appendToNewBlock whether to append data to a new block
+ * instead of the end of the last partial block
+ * @throws IOException IO failure
+ * @throws UnsupportedOperationException if the operation is unsupported
+ * (default).
+ * @return output stream.
+ */
+ public FSDataOutputStream append(Path f, int bufferSize,
+ Progressable progress, boolean appendToNewBlock) throws IOException {
+return append(f, bufferSize, progress);
Review Comment:
> TestDFSShell#testAppendToFileWithOptionN didn't contain a partial last
block, because the block size is `1024` and the file size in the UT is `1024 *
1024`, so the last block is not partial block. Maybe you can change the file
size to `1024 * 1024 + 10` to verify it again.
Hi @ZanderXu , there is a new configuration. And block size is not `1024`
but `128 * 1024 * 1024`. So the last block is partial.
> Improve appendToFile command: support appending on file with new block
> --
>
> Key: HDFS-16716
> URL: https://issues.apache.org/jira/browse/HDFS-16716
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: guojunhao
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> HDFS client DistributedFileSystem#append supports appending to a file with
> optional create flags.
> However, appendToFile command only supports the default create flag APPEND so
> that append on EC file without NEW_BLOCK create flag is not supported.
> Thus, it's necessary to improve appendToFile command by adding option n for
> it. Option n represents that use NEW_BLOCK create flag while appending file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16716) Improve appendToFile command: support appending on file with new block
[
https://issues.apache.org/jira/browse/HDFS-16716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623695#comment-17623695
]
ASF GitHub Bot commented on HDFS-16716:
---
ZanderXu commented on code in PR #4697:
URL: https://github.com/apache/hadoop/pull/4697#discussion_r100421
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java:
##
@@ -1544,6 +1544,39 @@ public FSDataOutputStream append(Path f, int bufferSize)
throws IOException {
public abstract FSDataOutputStream append(Path f, int bufferSize,
Progressable progress) throws IOException;
+ /**
+ * Append to an existing file (optional operation).
+ * @param f the existing file to be appended.
+ * @param appendToNewBlock whether to append data to a new block
+ * instead of the end of the last partial block
+ * @throws IOException IO failure
+ * @throws UnsupportedOperationException if the operation is unsupported
+ * (default).
+ * @return output stream.
+ */
+ public FSDataOutputStream append(Path f, boolean appendToNewBlock) throws
IOException {
+return append(f, getConf().getInt(IO_FILE_BUFFER_SIZE_KEY,
+IO_FILE_BUFFER_SIZE_DEFAULT), null, appendToNewBlock);
+ }
+
+ /**
+ * Append to an existing file (optional operation).
+ * This function is used for being overridden by some FileSystem like
DistributedFileSystem
+ * @param f the existing file to be appended.
+ * @param bufferSize the size of the buffer to be used.
+ * @param progress for reporting progress if it is not null.
+ * @param appendToNewBlock whether to append data to a new block
+ * instead of the end of the last partial block
+ * @throws IOException IO failure
+ * @throws UnsupportedOperationException if the operation is unsupported
+ * (default).
+ * @return output stream.
+ */
+ public FSDataOutputStream append(Path f, int bufferSize,
+ Progressable progress, boolean appendToNewBlock) throws IOException {
+return append(f, bufferSize, progress);
Review Comment:
Thanks @tomscut for your correction. I will review this PR carefully later.
> Improve appendToFile command: support appending on file with new block
> --
>
> Key: HDFS-16716
> URL: https://issues.apache.org/jira/browse/HDFS-16716
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: guojunhao
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> HDFS client DistributedFileSystem#append supports appending to a file with
> optional create flags.
> However, appendToFile command only supports the default create flag APPEND so
> that append on EC file without NEW_BLOCK create flag is not supported.
> Thus, it's necessary to improve appendToFile command by adding option n for
> it. Option n represents that use NEW_BLOCK create flag while appending file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16815) Error occurred in processing CacheManagerSection for xml parsing fsimage
[
https://issues.apache.org/jira/browse/HDFS-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623693#comment-17623693
]
ASF GitHub Bot commented on HDFS-16815:
---
hadoop-yetus commented on PR #5069:
URL: https://github.com/apache/hadoop/pull/5069#issuecomment-1290233790
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 57s | | trunk passed |
| +1 :green_heart: | compile | 1m 38s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 16s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 16s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 34s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 21s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 27s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 27s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 0s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5069/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 23 unchanged -
0 fixed = 25 total (was 23) |
| +1 :green_heart: | mvnsite | 1m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 28m 32s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 340m 45s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5069/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 54s | | The patch does not
generate ASF License warnings. |
| | | 462m 9s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5069/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5069 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 337381dc9756 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f09556ba6ad66502081dd80f30b6c2186ee91651 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
[jira] [Commented] (HDFS-16817) Remove useless DataNode lock related configuration
[
https://issues.apache.org/jira/browse/HDFS-16817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623588#comment-17623588
]
ASF GitHub Bot commented on HDFS-16817:
---
hadoop-yetus commented on PR #5072:
URL: https://github.com/apache/hadoop/pull/5072#issuecomment-1290063145
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 45s | | trunk passed |
| +1 :green_heart: | compile | 1m 43s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 34s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 16s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 47s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 47s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 44s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 30s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 29s | | the patch passed |
| +1 :green_heart: | compile | 1m 33s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 33s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 2s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 50s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 242m 41s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 3s | | The patch does not
generate ASF License warnings. |
| | | 364m 16s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5072/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5072 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 88d177eea7ad 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5bed5f3d3c6cbc8b639fa3066a23f04434de695a |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5072/1/testReport/ |
| Max. process+thread count | 3391 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5072/1/console |
| versions |
