[jira] [Commented] (HDFS-14762) "Path(Path/String parent, String child)" will fail when "child" contains ":"
[
https://issues.apache.org/jira/browse/HDFS-14762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109947#comment-17109947
]
hemanthboyina commented on HDFS-14762:
--
i think this issue also relates to IPV6
> "Path(Path/String parent, String child)" will fail when "child" contains ":"
>
>
> Key: HDFS-14762
> URL: https://issues.apache.org/jira/browse/HDFS-14762
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shixiong Zhu
>Priority: Major
> Attachments: HDFS-14762.001.patch, HDFS-14762.002.patch,
> HDFS-14762.003.patch, HDFS-14762.004.patch
>
>
> When the "child" parameter contains ":", "Path(Path/String parent, String
> child)" will throw the following exception:
> {code}
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI: ...
> {code}
> Not sure if this is a legit bug. But the following places will hit this error
> when seeing a Path with a file name containing ":":
> https://github.com/apache/hadoop/blob/f9029c4070e8eb046b403f5cb6d0a132c5d58448/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java#L101
> https://github.com/apache/hadoop/blob/f9029c4070e8eb046b403f5cb6d0a132c5d58448/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java#L270
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14762) "Path(Path/String parent, String child)" will fail when "child" contains ":"
[
https://issues.apache.org/jira/browse/HDFS-14762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109943#comment-17109943
]
Ayush Saxena commented on HDFS-14762:
-
bq. some file name used IPv6.
File Name used IPv6? What is the relation of name & IPv6?
If you want to follow this issue. Please follow at HADOOP-14829 This had same
problem, and may be getting a agreement and conclusion at HADOOP-3257
> "Path(Path/String parent, String child)" will fail when "child" contains ":"
>
>
> Key: HDFS-14762
> URL: https://issues.apache.org/jira/browse/HDFS-14762
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shixiong Zhu
>Priority: Major
> Attachments: HDFS-14762.001.patch, HDFS-14762.002.patch,
> HDFS-14762.003.patch, HDFS-14762.004.patch
>
>
> When the "child" parameter contains ":", "Path(Path/String parent, String
> child)" will throw the following exception:
> {code}
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI: ...
> {code}
> Not sure if this is a legit bug. But the following places will hit this error
> when seeing a Path with a file name containing ":":
> https://github.com/apache/hadoop/blob/f9029c4070e8eb046b403f5cb6d0a132c5d58448/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java#L101
> https://github.com/apache/hadoop/blob/f9029c4070e8eb046b403f5cb6d0a132c5d58448/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java#L270
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-14762) "Path(Path/String parent, String child)" will fail when "child" contains ":"
[
https://issues.apache.org/jira/browse/HDFS-14762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore reopened HDFS-14762:
---
We should handle this scenario. It is valid scenario.
We faced same problem when in some file name used IPv6.
> "Path(Path/String parent, String child)" will fail when "child" contains ":"
>
>
> Key: HDFS-14762
> URL: https://issues.apache.org/jira/browse/HDFS-14762
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shixiong Zhu
>Priority: Major
> Attachments: HDFS-14762.001.patch, HDFS-14762.002.patch,
> HDFS-14762.003.patch, HDFS-14762.004.patch
>
>
> When the "child" parameter contains ":", "Path(Path/String parent, String
> child)" will throw the following exception:
> {code}
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI: ...
> {code}
> Not sure if this is a legit bug. But the following places will hit this error
> when seeing a Path with a file name containing ":":
> https://github.com/apache/hadoop/blob/f9029c4070e8eb046b403f5cb6d0a132c5d58448/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java#L101
> https://github.com/apache/hadoop/blob/f9029c4070e8eb046b403f5cb6d0a132c5d58448/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java#L270
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15360) Update log output problems for ExternalSPSContext
[ https://issues.apache.org/jira/browse/HDFS-15360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bianqi updated HDFS-15360: -- Status: Open (was: Patch Available) > Update log output problems for ExternalSPSContext > - > > Key: HDFS-15360 > URL: https://issues.apache.org/jira/browse/HDFS-15360 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.2.1 >Reporter: bianqi >Assignee: bianqi >Priority: Major > Attachments: HDFS-15360.001.patch > > > Update log output problems for ExternalSPSContext. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15360) Update log output problems for ExternalSPSContext
[ https://issues.apache.org/jira/browse/HDFS-15360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bianqi resolved HDFS-15360. --- Resolution: Invalid > Update log output problems for ExternalSPSContext > - > > Key: HDFS-15360 > URL: https://issues.apache.org/jira/browse/HDFS-15360 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.2.1 >Reporter: bianqi >Assignee: bianqi >Priority: Major > Attachments: HDFS-15360.001.patch > > > Update log output problems for ExternalSPSContext. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15355) Make the default block storage policy ID configurable
[ https://issues.apache.org/jira/browse/HDFS-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109844#comment-17109844 ] Ayush Saxena commented on HDFS-15355: - Bunch of test failures seems related. > Make the default block storage policy ID configurable > - > > Key: HDFS-15355 > URL: https://issues.apache.org/jira/browse/HDFS-15355 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement, namenode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15355.001.patch, HDFS-15355.002.patch, > HDFS-15355.003.patch, HDFS-15355.004.patch, HDFS-15355.005.patch, > HDFS-15355.006.patch, HDFS-15355.007.patch > > > Make the default block storage policy ID configurable. Sometime we want to > use different storage policy ID from the startup of cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13183) Standby NameNode process getBlocks request to reduce Active load
[
https://issues.apache.org/jira/browse/HDFS-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109826#comment-17109826
]
Xiaoqiao He commented on HDFS-13183:
code rebase and trigger yetus again.
> Standby NameNode process getBlocks request to reduce Active load
>
>
> Key: HDFS-13183
> URL: https://issues.apache.org/jira/browse/HDFS-13183
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover, namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-13183-trunk.001.patch, HDFS-13183-trunk.002.patch,
> HDFS-13183-trunk.003.patch, HDFS-13183.004.patch, HDFS-13183.005.patch,
> HDFS-13183.006.patch, HDFS-13183.007.patch
>
>
> The performance of Active NameNode could be impact when {{Balancer}} requests
> #getBlocks, since query blocks of overly full DNs performance is extremely
> inefficient currently. The main reason is {{NameNodeRpcServer#getBlocks}}
> hold read lock for long time. In extreme case, all handlers of Active
> NameNode RPC server are occupied by one reader
> {{NameNodeRpcServer#getBlocks}} and other write operation calls, thus Active
> NameNode enter a state of false death for number of seconds even for minutes.
> The similar performance concerns of Balancer have reported by HDFS-9412,
> HDFS-7967, etc.
> If Standby NameNode can shoulder #getBlocks heavy burden, it could speed up
> the progress of balancing and reduce performance impact to Active NameNode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13183) Standby NameNode process getBlocks request to reduce Active load
[
https://issues.apache.org/jira/browse/HDFS-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-13183:
---
Attachment: HDFS-13183.007.patch
> Standby NameNode process getBlocks request to reduce Active load
>
>
> Key: HDFS-13183
> URL: https://issues.apache.org/jira/browse/HDFS-13183
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover, namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-13183-trunk.001.patch, HDFS-13183-trunk.002.patch,
> HDFS-13183-trunk.003.patch, HDFS-13183.004.patch, HDFS-13183.005.patch,
> HDFS-13183.006.patch, HDFS-13183.007.patch
>
>
> The performance of Active NameNode could be impact when {{Balancer}} requests
> #getBlocks, since query blocks of overly full DNs performance is extremely
> inefficient currently. The main reason is {{NameNodeRpcServer#getBlocks}}
> hold read lock for long time. In extreme case, all handlers of Active
> NameNode RPC server are occupied by one reader
> {{NameNodeRpcServer#getBlocks}} and other write operation calls, thus Active
> NameNode enter a state of false death for number of seconds even for minutes.
> The similar performance concerns of Balancer have reported by HDFS-9412,
> HDFS-7967, etc.
> If Standby NameNode can shoulder #getBlocks heavy burden, it could speed up
> the progress of balancing and reduce performance impact to Active NameNode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15293) Relax the condition for accepting a fsimage when receiving a checkpoint
[ https://issues.apache.org/jira/browse/HDFS-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109727#comment-17109727 ] Akira Ajisaka commented on HDFS-15293: -- +1, thanks [~vagarychen] for the update. > Relax the condition for accepting a fsimage when receiving a checkpoint > > > Key: HDFS-15293 > URL: https://issues.apache.org/jira/browse/HDFS-15293 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Critical > Labels: multi-sbnn, release-blocker > Attachments: HDFS-15293.001.patch, HDFS-15293.002.patch > > > HDFS-12979 introduced the logic that, if ANN sees consecutive fs image upload > from Standby with a small delta comparing to previous fsImage. ANN would > reject this image. This is to avoid overly frequent fsImage in case of when > there are multiple Standby node. However this check could be too stringent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15360) Update log output problems for ExternalSPSContext
[
https://issues.apache.org/jira/browse/HDFS-15360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109640#comment-17109640
]
Hadoop QA commented on HDFS-15360:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 22m
5s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 18m
11s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
2s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 15m
59s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}109m 44s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.namenode.ha.TestHAAppend |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-HDFS-Build/29316/artifact/out/Dockerfile
|
| JIRA Issue | HDFS-15360 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13003225/HDFS-15360.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux c636b1402b62 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / a3809d20230 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| mvninstall |
https:/
[jira] [Updated] (HDFS-15360) Update log output problems for ExternalSPSContext
[ https://issues.apache.org/jira/browse/HDFS-15360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bianqi updated HDFS-15360: -- Status: Patch Available (was: Open) > Update log output problems for ExternalSPSContext > - > > Key: HDFS-15360 > URL: https://issues.apache.org/jira/browse/HDFS-15360 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.2.1 >Reporter: bianqi >Assignee: bianqi >Priority: Major > Attachments: HDFS-15360.001.patch > > > Update log output problems for ExternalSPSContext. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15360) Update log output problems for ExternalSPSContext
[ https://issues.apache.org/jira/browse/HDFS-15360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bianqi updated HDFS-15360: -- Attachment: HDFS-15360.001.patch > Update log output problems for ExternalSPSContext > - > > Key: HDFS-15360 > URL: https://issues.apache.org/jira/browse/HDFS-15360 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.2.1 >Reporter: bianqi >Assignee: bianqi >Priority: Major > Attachments: HDFS-15360.001.patch > > > Update log output problems for ExternalSPSContext. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15360) Update log output problems for ExternalSPSContext
bianqi created HDFS-15360: - Summary: Update log output problems for ExternalSPSContext Key: HDFS-15360 URL: https://issues.apache.org/jira/browse/HDFS-15360 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.2.1 Reporter: bianqi Assignee: bianqi Update log output problems for ExternalSPSContext. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15355) Make the default block storage policy ID configurable
[
https://issues.apache.org/jira/browse/HDFS-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109543#comment-17109543
]
Hadoop QA commented on HDFS-15355:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
56s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 20 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 24m
56s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
22s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 24m
10s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
6s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
46s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 15s{color} | {color:orange} root: The patch generated 1 new + 1462 unchanged
- 1 fixed = 1463 total (was 1463) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 16m
6s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
11s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 12m 46s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 51s{color}
| {color:red} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 30s{color}
| {color:red} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
40s{color} | {color:green} hadoop-fs2img in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:bla
[jira] [Updated] (HDFS-13639) SlotReleaser is not fast enough
[
https://issues.apache.org/jira/browse/HDFS-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-13639:
---
Attachment: image-2020-05-17-23-53-41-115.png
> SlotReleaser is not fast enough
> ---
>
> Key: HDFS-13639
> URL: https://issues.apache.org/jira/browse/HDFS-13639
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 2.4.0, 2.6.0, 3.0.2
> Environment: 1. YCSB:
> {color:#00} recordcount=20
> fieldcount=1
> fieldlength=1000
> operationcount=1000
>
> workload=com.yahoo.ycsb.workloads.CoreWorkload
>
> table=ycsb-test
> columnfamily=C
> readproportion=1
> updateproportion=0
> insertproportion=0
> scanproportion=0
>
> maxscanlength=0
> requestdistribution=zipfian
>
> # default
> readallfields=true
> writeallfields=true
> scanlengthdistribution=constan{color}
> {color:#00}2. datanode:{color}
> -Xmx2048m -Xms2048m -Xmn1024m -XX:MaxDirectMemorySize=1024m
> -XX:MaxPermSize=256m -Xloggc:$run_dir/stdout/datanode_gc_${start_time}.log
> -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=$log_dir -XX:+PrintGCApplicationStoppedTime
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80
> -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSParallelRemarkEnabled
> -XX:+CMSClassUnloadingEnabled -XX:CMSMaxAbortablePrecleanTime=1
> -XX:+CMSScavengeBeforeRemark -XX:+PrintPromotionFailure
> -XX:+CMSConcurrentMTEnabled -XX:+ExplicitGCInvokesConcurrent
> -XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking
> -verbose:gc -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps
> {color:#00}3. regionserver:{color}
> {color:#00}-Xmx10g -Xms10g -XX:MaxDirectMemorySize=10g
> -XX:MaxGCPauseMillis=150 -XX:MaxTenuringThreshold=2
> -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=5
> -Xloggc:$run_dir/stdout/regionserver_gc_${start_time}.log -Xss256k
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$log_dir -verbose:gc
> -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps -XX:+PrintAdaptiveSizePolicy
> -XX:+PrintTenuringDistribution -XX:+PrintSafepointStatistics
> -XX:PrintSafepointStatisticsCount=1 -XX:PrintFLSStatistics=1
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=100 -XX:GCLogFileSize=128m
> -XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking
> -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=65
> -XX:+ParallelRefProcEnabled -XX:ConcGCThreads=4 -XX:ParallelGCThreads=16
> -XX:G1HeapRegionSize=32m -XX:G1MixedGCCountTarget=64
> -XX:G1OldCSetRegionThresholdPercent=5{color}
> {color:#00}block cache is disabled:{color}{color:#00}
> hbase.bucketcache.size
> 0.9
> {color}
>
>Reporter: Gang Xie
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-13639-2.4.diff, HDFS-13639.001.patch,
> HDFS-13639.002.patch, ShortCircuitCache_new_slotReleaser.diff,
> image-2020-05-17-23-53-41-115.png, perf_after_improve_SlotReleaser.png,
> perf_before_improve_SlotReleaser.png
>
>
> When test the performance of the ShortCircuit Read of the HDFS with YCSB, we
> find that SlotReleaser of the ShortCircuitCache has some performance issue.
> The problem is that, the qps of the slot releasing could only reach to 1000+
> while the qps of the slot allocating is ~3000. This means that the replica
> info on datanode could not be released in time, which causes a lot of GCs and
> finally full GCs.
>
> The fireflame graph shows that SlotReleaser spends a lot of time to do domain
> socket connecting and throw/catching the exception when close the domain
> socket and its streams. It doesn't make any sense to do the connecting and
> closing each time. Each time when we connect to the domain socket, Datanode
> allocates a new thread to free the slot. There are a lot of initializing
> work, and it's costly. We need reuse the domain socket.
>
> After switch to reuse the domain socket(see diff attached), we get great
> improvement(see the perf):
> # without reusing the domain socket, the get qps of the YCSB getting worse
> and worse, and after about 45 mins, full GC starts. When we reuse the domain
> socket, no full GC found, and the stress test could be finished smoothly, the
> qps of allocating and releasing match.
> # Due to the datanode young GC, without the improvement, the YCSB get qps is
> even smaller than the one with the improvement, ~3700 VS ~4200.
> The diff is against 2.4, and I think this issue exists till latest version. I
> doesn't have test env with 2.7 and higher version.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issue
[jira] [Commented] (HDFS-15082) RBF: Check each component length of destination path when add/update mount entry
[ https://issues.apache.org/jira/browse/HDFS-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109498#comment-17109498 ] Hudson commented on HDFS-15082: --- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #18263 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18263/]) HDFS-15082. RBF: Check each component length of destination path when (ayushsaxena: rev a3809d202300ce39c75e909fbc4640635dc334bc) * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterMountTable.java * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RBFConfigKeys.java * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterAdminServer.java * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/resources/hdfs-rbf-default.xml > RBF: Check each component length of destination path when add/update mount > entry > > > Key: HDFS-15082 > URL: https://issues.apache.org/jira/browse/HDFS-15082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Fix For: 3.4.0 > > Attachments: HDFS-15082.001.patch, HDFS-15082.002.patch, > HDFS-15082.003.patch, HDFS-15082.004.patch, HDFS-15082.005.patch > > > When add/update mount entry, each component length of destination path could > exceed filesystem path component length limit, reference to > `dfs.namenode.fs-limits.max-component-length` of NameNode. So we should check > each component length of destination path when add/update mount entry at > Router side. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15082) RBF: Check each component length of destination path when add/update mount entry
[ https://issues.apache.org/jira/browse/HDFS-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109495#comment-17109495 ] Ayush Saxena commented on HDFS-15082: - Committed to trunk. Thanx [~hexiaoqiao] for the contribution and [~elgoiri] for the review!!! > RBF: Check each component length of destination path when add/update mount > entry > > > Key: HDFS-15082 > URL: https://issues.apache.org/jira/browse/HDFS-15082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-15082.001.patch, HDFS-15082.002.patch, > HDFS-15082.003.patch, HDFS-15082.004.patch, HDFS-15082.005.patch > > > When add/update mount entry, each component length of destination path could > exceed filesystem path component length limit, reference to > `dfs.namenode.fs-limits.max-component-length` of NameNode. So we should check > each component length of destination path when add/update mount entry at > Router side. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15082) RBF: Check each component length of destination path when add/update mount entry
[ https://issues.apache.org/jira/browse/HDFS-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15082: Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > RBF: Check each component length of destination path when add/update mount > entry > > > Key: HDFS-15082 > URL: https://issues.apache.org/jira/browse/HDFS-15082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Fix For: 3.4.0 > > Attachments: HDFS-15082.001.patch, HDFS-15082.002.patch, > HDFS-15082.003.patch, HDFS-15082.004.patch, HDFS-15082.005.patch > > > When add/update mount entry, each component length of destination path could > exceed filesystem path component length limit, reference to > `dfs.namenode.fs-limits.max-component-length` of NameNode. So we should check > each component length of destination path when add/update mount entry at > Router side. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15355) Make the default block storage policy ID configurable
[ https://issues.apache.org/jira/browse/HDFS-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109476#comment-17109476 ] Yang Yun commented on HDFS-15355: - [~ayushtkn], I'm sorry, I uploaded a wrong file. HDFS-15355.007.patch is uploaded. > Make the default block storage policy ID configurable > - > > Key: HDFS-15355 > URL: https://issues.apache.org/jira/browse/HDFS-15355 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement, namenode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15355.001.patch, HDFS-15355.002.patch, > HDFS-15355.003.patch, HDFS-15355.004.patch, HDFS-15355.005.patch, > HDFS-15355.006.patch, HDFS-15355.007.patch > > > Make the default block storage policy ID configurable. Sometime we want to > use different storage policy ID from the startup of cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15355) Make the default block storage policy ID configurable
[ https://issues.apache.org/jira/browse/HDFS-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15355: Attachment: (was: HDFS-15207.005.patch) > Make the default block storage policy ID configurable > - > > Key: HDFS-15355 > URL: https://issues.apache.org/jira/browse/HDFS-15355 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement, namenode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15355.001.patch, HDFS-15355.002.patch, > HDFS-15355.003.patch, HDFS-15355.004.patch, HDFS-15355.005.patch, > HDFS-15355.006.patch, HDFS-15355.007.patch > > > Make the default block storage policy ID configurable. Sometime we want to > use different storage policy ID from the startup of cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15355) Make the default block storage policy ID configurable
[ https://issues.apache.org/jira/browse/HDFS-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15355: Status: Open (was: Patch Available) > Make the default block storage policy ID configurable > - > > Key: HDFS-15355 > URL: https://issues.apache.org/jira/browse/HDFS-15355 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement, namenode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15207.005.patch, HDFS-15355.001.patch, > HDFS-15355.002.patch, HDFS-15355.003.patch, HDFS-15355.004.patch, > HDFS-15355.005.patch, HDFS-15355.006.patch > > > Make the default block storage policy ID configurable. Sometime we want to > use different storage policy ID from the startup of cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15355) Make the default block storage policy ID configurable
[ https://issues.apache.org/jira/browse/HDFS-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15355: Attachment: HDFS-15355.007.patch Status: Patch Available (was: Open) > Make the default block storage policy ID configurable > - > > Key: HDFS-15355 > URL: https://issues.apache.org/jira/browse/HDFS-15355 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement, namenode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15207.005.patch, HDFS-15355.001.patch, > HDFS-15355.002.patch, HDFS-15355.003.patch, HDFS-15355.004.patch, > HDFS-15355.005.patch, HDFS-15355.006.patch, HDFS-15355.007.patch > > > Make the default block storage policy ID configurable. Sometime we want to > use different storage policy ID from the startup of cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15355) Make the default block storage policy ID configurable
[
https://issues.apache.org/jira/browse/HDFS-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109456#comment-17109456
]
Hadoop QA commented on HDFS-15355:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
7s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 27m
33s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 21m
54s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
38s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 52s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 501 unchanged - 0 fixed = 502 total (was 501) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 18m
13s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}113m 36s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}202m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-HDFS-Build/29313/artifact/out/Dockerfile
|
| JIRA Issue | HDFS-15355 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13003177/HDFS-15207.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux a13dcdd217df 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 6e416a83d1e |
| Default Java | Private Build-
[jira] [Commented] (HDFS-15355) Make the default block storage policy ID configurable
[ https://issues.apache.org/jira/browse/HDFS-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109454#comment-17109454 ] Ayush Saxena commented on HDFS-15355: - [~hadoop_yangyun] seems you haven't uploaded the v007 patch? > Make the default block storage policy ID configurable > - > > Key: HDFS-15355 > URL: https://issues.apache.org/jira/browse/HDFS-15355 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement, namenode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15207.005.patch, HDFS-15355.001.patch, > HDFS-15355.002.patch, HDFS-15355.003.patch, HDFS-15355.004.patch, > HDFS-15355.005.patch, HDFS-15355.006.patch > > > Make the default block storage policy ID configurable. Sometime we want to > use different storage policy ID from the startup of cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15082) RBF: Check each component length of destination path when add/update mount entry
[ https://issues.apache.org/jira/browse/HDFS-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109453#comment-17109453 ] Ayush Saxena commented on HDFS-15082: - v005 LGTM +1 > RBF: Check each component length of destination path when add/update mount > entry > > > Key: HDFS-15082 > URL: https://issues.apache.org/jira/browse/HDFS-15082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-15082.001.patch, HDFS-15082.002.patch, > HDFS-15082.003.patch, HDFS-15082.004.patch, HDFS-15082.005.patch > > > When add/update mount entry, each component length of destination path could > exceed filesystem path component length limit, reference to > `dfs.namenode.fs-limits.max-component-length` of NameNode. So we should check > each component length of destination path when add/update mount entry at > Router side. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15355) Make the default block storage policy ID configurable
[ https://issues.apache.org/jira/browse/HDFS-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109403#comment-17109403 ] Yang Yun commented on HDFS-15355: - Update to HDFS-15355.007.patch for checkstyle issues. Current truck code build failed with below failure even without any change in my local setup. I'm not sure the reason. [INFO] Apache Hadoop Client Packaging Invariants for Test . FAILURE [ 0.167 s] > Make the default block storage policy ID configurable > - > > Key: HDFS-15355 > URL: https://issues.apache.org/jira/browse/HDFS-15355 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement, namenode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15207.005.patch, HDFS-15355.001.patch, > HDFS-15355.002.patch, HDFS-15355.003.patch, HDFS-15355.004.patch, > HDFS-15355.005.patch, HDFS-15355.006.patch > > > Make the default block storage policy ID configurable. Sometime we want to > use different storage policy ID from the startup of cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15355) Make the default block storage policy ID configurable
[ https://issues.apache.org/jira/browse/HDFS-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15355: Status: Open (was: Patch Available) > Make the default block storage policy ID configurable > - > > Key: HDFS-15355 > URL: https://issues.apache.org/jira/browse/HDFS-15355 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement, namenode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15207.005.patch, HDFS-15355.001.patch, > HDFS-15355.002.patch, HDFS-15355.003.patch, HDFS-15355.004.patch, > HDFS-15355.005.patch, HDFS-15355.006.patch > > > Make the default block storage policy ID configurable. Sometime we want to > use different storage policy ID from the startup of cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15355) Make the default block storage policy ID configurable
[ https://issues.apache.org/jira/browse/HDFS-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15355: Attachment: HDFS-15207.005.patch Status: Patch Available (was: Open) > Make the default block storage policy ID configurable > - > > Key: HDFS-15355 > URL: https://issues.apache.org/jira/browse/HDFS-15355 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement, namenode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15207.005.patch, HDFS-15355.001.patch, > HDFS-15355.002.patch, HDFS-15355.003.patch, HDFS-15355.004.patch, > HDFS-15355.005.patch, HDFS-15355.006.patch > > > Make the default block storage policy ID configurable. Sometime we want to > use different storage policy ID from the startup of cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15355) Make the default block storage policy ID configurable
[ https://issues.apache.org/jira/browse/HDFS-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109385#comment-17109385 ] Ayush Saxena edited comment on HDFS-15355 at 5/17/20, 8:05 AM: --- Can you give a check to the test failures and checkstyle issues. [~elgoiri] these variables were "public" and available in hdfs-client, Is removing them safe/allowed. Anyone using can get affected? I am not sure about it. Can you help check once. was (Author: ayushtkn): Can you give a check to the test failures and checkstyle issues > Make the default block storage policy ID configurable > - > > Key: HDFS-15355 > URL: https://issues.apache.org/jira/browse/HDFS-15355 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement, namenode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15355.001.patch, HDFS-15355.002.patch, > HDFS-15355.003.patch, HDFS-15355.004.patch, HDFS-15355.005.patch, > HDFS-15355.006.patch > > > Make the default block storage policy ID configurable. Sometime we want to > use different storage policy ID from the startup of cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15355) Make the default block storage policy ID configurable
[ https://issues.apache.org/jira/browse/HDFS-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109385#comment-17109385 ] Ayush Saxena commented on HDFS-15355: - Can you give a check to the test failures and checkstyle issues > Make the default block storage policy ID configurable > - > > Key: HDFS-15355 > URL: https://issues.apache.org/jira/browse/HDFS-15355 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement, namenode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15355.001.patch, HDFS-15355.002.patch, > HDFS-15355.003.patch, HDFS-15355.004.patch, HDFS-15355.005.patch, > HDFS-15355.006.patch > > > Make the default block storage policy ID configurable. Sometime we want to > use different storage policy ID from the startup of cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15359) EC: Allow closing a file with committed blocks
[
https://issues.apache.org/jira/browse/HDFS-15359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109384#comment-17109384
]
Ayush Saxena commented on HDFS-15359:
-
Jenkins warnings doesn't seems related.
Pls Review!!!
> EC: Allow closing a file with committed blocks
> --
>
> Key: HDFS-15359
> URL: https://issues.apache.org/jira/browse/HDFS-15359
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15359-01.patch, HDFS-15359-02.patch,
> HDFS-15359-03.patch
>
>
> Presently, {{dfs.namenode.file.close.num-committed-allowed}} is ignored in
> case of EC blocks. But in case of heavy loads, IBR's from Datanode may get
> delayed and cause the file write to fail. So, can allow EC files to close
> with blocks in committed state as REP files
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
