[jira] [Commented] (HDFS-15463) Add a tool to validate FsImage
[ https://issues.apache.org/jira/browse/HDFS-15463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17160940#comment-17160940 ] Hudson commented on HDFS-15463: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18451 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18451/]) HDFS-15463. Add a tool to validate FsImage. (#2140) (github: rev 2cec50cf1657672e14541717b8222cecc3ad5dd0) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs.cmd * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FsImageValidation.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsImageValidation.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReferenceValidation.java > Add a tool to validate FsImage > -- > > Key: HDFS-15463 > URL: https://issues.apache.org/jira/browse/HDFS-15463 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > Fix For: 3.4.0 > > Attachments: FsImageValidation20200709.patch, > FsImageValidation20200712.patch, FsImageValidation20200714.patch, > FsImageValidation20200715.patch, FsImageValidation20200715b.patch, > FsImageValidation20200715c.patch, FsImageValidation20200717b.patch, > FsImageValidation20200718.patch, HDFS-15463.000.patch > > > Due to some snapshot related bugs, a fsimage may become corrupted. Using a > corrupted fsimage may further result in data loss. > In some cases, we found that reference counts are incorrect in some corrupted > FsImage. One of the goals of the validation tool is to check reference > counts. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15463) Add a tool to validate FsImage
[ https://issues.apache.org/jira/browse/HDFS-15463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee resolved HDFS-15463. Fix Version/s: 3.4.0 Resolution: Fixed > Add a tool to validate FsImage > -- > > Key: HDFS-15463 > URL: https://issues.apache.org/jira/browse/HDFS-15463 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > Fix For: 3.4.0 > > Attachments: FsImageValidation20200709.patch, > FsImageValidation20200712.patch, FsImageValidation20200714.patch, > FsImageValidation20200715.patch, FsImageValidation20200715b.patch, > FsImageValidation20200715c.patch, FsImageValidation20200717b.patch, > FsImageValidation20200718.patch, HDFS-15463.000.patch > > > Due to some snapshot related bugs, a fsimage may become corrupted. Using a > corrupted fsimage may further result in data loss. > In some cases, we found that reference counts are incorrect in some corrupted > FsImage. One of the goals of the validation tool is to check reference > counts. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15098) Add SM4 encryption method for HDFS
[
https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17160911#comment-17160911
]
Hadoop QA commented on HDFS-15098:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} prototool {color} | {color:blue} 0m
0s{color} | {color:blue} prototool was not available. {color} |
| {color:blue}0{color} | {color:blue} markdownlint {color} | {color:blue} 0m
0s{color} | {color:blue} markdownlint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
22m 35s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
25s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
39s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
12s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 2
extant findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
37s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 4 extant
findbugs warnings. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 18m 40s{color} |
{color:red} root generated 17 new + 145 unchanged - 17 fixed = 162 total (was
162) {color} |
| {color:green}+1{color} | {color:green} golang {color} | {color:green} 18m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 18m 40s{color}
| {color:red} root generated 4 new + 1948 unchanged - 0 fixed = 1952 total (was
1948) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 17s{color} | {color:orange} root: The patch generated 4 new + 229 unchanged
- 8 fixed = 233 total (was 237) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 4 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit
[jira] [Commented] (HDFS-15474) HttpFS: WebHdfsFileSystem cannot renew an expired delegation token from HttpFS response
[
https://issues.apache.org/jira/browse/HDFS-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17160890#comment-17160890
]
Takanobu Asanuma commented on HDFS-15474:
-
It seems an interesting idea. Actually, our cluster has multiple HttpFS
servers, and they share delegation tokens using a ZooKeeper cluster.
> HttpFS: WebHdfsFileSystem cannot renew an expired delegation token from
> HttpFS response
> ---
>
> Key: HDFS-15474
> URL: https://issues.apache.org/jira/browse/HDFS-15474
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
>
> When clients use WebHdfsFileSystem for HttpFS, they cannot renew expired
> delegation tokens with the following error.
> {noformat}
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: token
> (owner=..., renewer=..., realUser=..., issueDate=..., maxDate=...,
> sequenceNumber=..., masterKeyId=...) is expired
> at
> org.apache.hadoop.hdfs.web.JsonUtilClient.toRemoteException(JsonUtilClient.java:89)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:509)
> ...
> {noformat}
> When using WebHdfsFileSystem for NameNode, it succeeds. This is because the
> response of HttpFS is different from its of NameNode. We should fix the
> response of HttpFS.
> This issue is reported by Masayuki Yatagawa.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zZtai updated HDFS-15098: - Attachment: HDFS-15098.009.patch Status: Patch Available (was: Open) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch, > HDFS-15098.009.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zZtai updated HDFS-15098: - Attachment: (was: HDFS-15098.009.patch) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zZtai updated HDFS-15098: - Status: Open (was: Patch Available) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zZtai updated HDFS-15098: - Attachment: HDFS-15098.009.patch Status: Patch Available (was: Open) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch, > HDFS-15098.009.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zZtai updated HDFS-15098: - Attachment: (was: HDFS-15098.009.patch) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zZtai updated HDFS-15098: - Status: Open (was: Patch Available) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17160852#comment-17160852 ] huangtianhua commented on HDFS-15025: - [~liuml07] Would you please help to review this, thanks very much. > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: hadoop_hdfs_hw >Priority: Major > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zZtai updated HDFS-15098: - Attachment: HDFS-15098.009.patch Status: Patch Available (was: Open) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch, > HDFS-15098.009.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zZtai updated HDFS-15098: - Status: Open (was: Patch Available) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch, > HDFS-15098.009.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zZtai updated HDFS-15098: - Attachment: (was: HDFS-15098.009.patch) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch, > HDFS-15098.009.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15476) Make AsyncStream class' executor_ member private
[ https://issues.apache.org/jira/browse/HDFS-15476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17160782#comment-17160782 ] Hudson commented on HDFS-15476: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18450 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18450/]) HDFS-15476 Make AsyncStream executor private (#2151) (github: rev 4101b0c0edab62a2f9fdbeb3071dc602fac45961) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/async_stream.h > Make AsyncStream class' executor_ member private > > > Key: HDFS-15476 > URL: https://issues.apache.org/jira/browse/HDFS-15476 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build, libhdfs++ >Reporter: Suraj Naik >Assignee: Suraj Naik >Priority: Minor > Fix For: 3.4.0 > > > As part of [HDFS-15385|https://issues.apache.org/jira/browse/HDFS-15385] the > boost library was upgraded. > The AsyncStream class has a getter function which returns the executor. > Keeping the executor member public makes the getter function's role > pointless. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15476) Make AsyncStream class' executor_ member private
[ https://issues.apache.org/jira/browse/HDFS-15476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore resolved HDFS-15476. --- Resolution: Fixed Thanks for contribution [~Suraj Naik] > Make AsyncStream class' executor_ member private > > > Key: HDFS-15476 > URL: https://issues.apache.org/jira/browse/HDFS-15476 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build, libhdfs++ >Reporter: Suraj Naik >Assignee: Suraj Naik >Priority: Minor > Fix For: 3.4.0 > > > As part of [HDFS-15385|https://issues.apache.org/jira/browse/HDFS-15385] the > boost library was upgraded. > The AsyncStream class has a getter function which returns the executor. > Keeping the executor member public makes the getter function's role > pointless. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15479) Ordered snapshot deletion: make it a configurable feature
[ https://issues.apache.org/jira/browse/HDFS-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17160772#comment-17160772 ] Tsz-wo Sze edited comment on HDFS-15479 at 7/19/20, 5:52 PM: - h15479_20200719.patch: add a conf Will add a test. was (Author: szetszwo): h15479_20200719.patch: add a conf > Ordered snapshot deletion: make it a configurable feature > - > > Key: HDFS-15479 > URL: https://issues.apache.org/jira/browse/HDFS-15479 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > Attachments: h15479_20200719.patch > > > Ordered snapshot deletion is a configurable feature. In this JIRA, a conf is > added. > When the feature is enabled, only the earliest snapshot can be deleted. For > deleting the non-earliest snapshots, the behavior is temporarily changed to > throwing an exception in this JIRA. In HDFS-15480, the behavior of deleting > the non-earliest snapshots will be changed to marking them as deleted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15479) Ordered snapshot deletion: make it a configurable feature
[ https://issues.apache.org/jira/browse/HDFS-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz-wo Sze updated HDFS-15479: -- Attachment: h15479_20200719.patch > Ordered snapshot deletion: make it a configurable feature > - > > Key: HDFS-15479 > URL: https://issues.apache.org/jira/browse/HDFS-15479 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > Attachments: h15479_20200719.patch > > > Ordered snapshot deletion is a configurable feature. In this JIRA, a conf is > added. > When the feature is enabled, only the earliest snapshot can be deleted. For > deleting the non-earliest snapshots, the behavior is temporarily changed to > throwing an exception in this JIRA. In HDFS-15480, the behavior of deleting > the non-earliest snapshots will be changed to marking them as deleted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15479) Ordered snapshot deletion: make it a configurable feature
[ https://issues.apache.org/jira/browse/HDFS-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz-wo Sze updated HDFS-15479: -- Attachment: (was: h15479_20200719.patch) > Ordered snapshot deletion: make it a configurable feature > - > > Key: HDFS-15479 > URL: https://issues.apache.org/jira/browse/HDFS-15479 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > Attachments: h15479_20200719.patch > > > Ordered snapshot deletion is a configurable feature. In this JIRA, a conf is > added. > When the feature is enabled, only the earliest snapshot can be deleted. For > deleting the non-earliest snapshots, the behavior is temporarily changed to > throwing an exception in this JIRA. In HDFS-15480, the behavior of deleting > the non-earliest snapshots will be changed to marking them as deleted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15479) Ordered snapshot deletion: make it a configurable feature
[ https://issues.apache.org/jira/browse/HDFS-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17160772#comment-17160772 ] Tsz-wo Sze commented on HDFS-15479: --- h15479_20200719.patch: add a conf > Ordered snapshot deletion: make it a configurable feature > - > > Key: HDFS-15479 > URL: https://issues.apache.org/jira/browse/HDFS-15479 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > Attachments: h15479_20200719.patch > > > Ordered snapshot deletion is a configurable feature. In this JIRA, a conf is > added. > When the feature is enabled, only the earliest snapshot can be deleted. For > deleting the non-earliest snapshots, the behavior is temporarily changed to > throwing an exception in this JIRA. In HDFS-15480, the behavior of deleting > the non-earliest snapshots will be changed to marking them as deleted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15479) Ordered snapshot deletion: make it a configurable feature
[ https://issues.apache.org/jira/browse/HDFS-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz-wo Sze updated HDFS-15479: -- Attachment: h15479_20200719.patch > Ordered snapshot deletion: make it a configurable feature > - > > Key: HDFS-15479 > URL: https://issues.apache.org/jira/browse/HDFS-15479 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > Attachments: h15479_20200719.patch > > > Ordered snapshot deletion is a configurable feature. In this JIRA, a conf is > added. > When the feature is enabled, only the earliest snapshot can be deleted. For > deleting the non-earliest snapshots, the behavior is temporarily changed to > throwing an exception in this JIRA. In HDFS-15480, the behavior of deleting > the non-earliest snapshots will be changed to marking them as deleted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15483) Ordered snapshot deletion: Disallow rename between two snapshottable directories
Tsz-wo Sze created HDFS-15483: - Summary: Ordered snapshot deletion: Disallow rename between two snapshottable directories Key: HDFS-15483 URL: https://issues.apache.org/jira/browse/HDFS-15483 Project: Hadoop HDFS Issue Type: Sub-task Components: snapshots Reporter: Tsz-wo Sze With the ordered snapshot deletion feature, only the *earliest* snapshot can be actually deleted from the file system. If renaming between snapshottable directories is allowed, only the earliest snapshot among all the snapshottable directories can be actually deleted. In such case, individual snapshottable directory may not be able to free up the resources by itself. Therefore, we propose disallowing renaming between snapshottable directories in this JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15477) Enforce ordered snapshot deletion
[ https://issues.apache.org/jira/browse/HDFS-15477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz-wo Sze updated HDFS-15477: -- Description: Snapshot deletion has caused a few bugs earlier such as HDFS-13101 and HDFS-15313. In this JIRA, we propose a configurable feature for enforcing ordered snapshot deletion -- only the earliest snapshot is actually deleted from the file system. The other snapshots are only marked as deleted. They will be actually deleted from the file system until all the earlier snapshots are deleted. The reason of enforcing ordered snapshot deletion is based on the observation that the logic of deleting the earliest snapshot is much simpler since the prior snapshot does not exist. All the previous bugs are caused by removing inodes from the prior snapshot. One drawback of ordered snapshot deletion is that the resources are not yet released for deleting the non-earliest snapshots since they are only marked as deleted but not actually deleted. was: Snapshot deletion has caused a few bugs earlier such as HDFS-13101 and HDFS-15313. In this JIRA, we propose enforcing ordered snapshot deletion -- only the earliest snapshot is actually deleted from the file system. The other snapshots are only marked as deleted. They will be actually deleted from the file system until all the earlier snapshots are deleted. The reason of enforcing ordered snapshot deletion is based on the observation that the logic of deleting the earliest snapshot is much simpler since the prior snapshot does not exist. All the previous bugs are caused by removing inodes from the prior snapshot. One drawback of ordered snapshot deletion is that the resources are not yet released for deleting the non-earliest snapshots since they are only marked as deleted but not actually deleted. > Enforce ordered snapshot deletion > - > > Key: HDFS-15477 > URL: https://issues.apache.org/jira/browse/HDFS-15477 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > > Snapshot deletion has caused a few bugs earlier such as HDFS-13101 and > HDFS-15313. In this JIRA, we propose a configurable feature for enforcing > ordered snapshot deletion -- only the earliest snapshot is actually deleted > from the file system. The other snapshots are only marked as deleted. They > will be actually deleted from the file system until all the earlier snapshots > are deleted. > The reason of enforcing ordered snapshot deletion is based on the observation > that the logic of deleting the earliest snapshot is much simpler since the > prior snapshot does not exist. All the previous bugs are caused by removing > inodes from the prior snapshot. > One drawback of ordered snapshot deletion is that the resources are not yet > released for deleting the non-earliest snapshots since they are only marked > as deleted but not actually deleted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15480) Ordered snapshot deletion: record snapshot deletion in XAttr
[ https://issues.apache.org/jira/browse/HDFS-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz-wo Sze updated HDFS-15480: -- Description: In this JIRA, the behavior of deleting the non-earliest snapshots will be changed to marking them as deleted in XAttr but not actually deleting them. Note that # The marked-for-deletion snapshots will be garbage collected later on; see HDFS-15481. # The marked-for-deletion snapshots will be hided from users; see HDFS-15482. was: In this JIRA, the behavior of deleting the non-earliest snapshots will be changed to marking them as deleted in XAttr but not actually deleting them. > Ordered snapshot deletion: record snapshot deletion in XAttr > > > Key: HDFS-15480 > URL: https://issues.apache.org/jira/browse/HDFS-15480 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz-wo Sze >Priority: Major > > In this JIRA, the behavior of deleting the non-earliest snapshots will be > changed to marking them as deleted in XAttr but not actually deleting them. > Note that > # The marked-for-deletion snapshots will be garbage collected later on; see > HDFS-15481. > # The marked-for-deletion snapshots will be hided from users; see HDFS-15482. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15482) Ordered snapshot deletion: hide the deleted snapshots from users
[ https://issues.apache.org/jira/browse/HDFS-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz-wo Sze updated HDFS-15482: -- Component/s: snapshots > Ordered snapshot deletion: hide the deleted snapshots from users > > > Key: HDFS-15482 > URL: https://issues.apache.org/jira/browse/HDFS-15482 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz-wo Sze >Priority: Major > > In HDFS-15480, the behavior of deleting the non-earliest snapshots is > changed to marking them as deleted in XAttr but not actually deleting them. > The users are still able to access the these snapshots as usual. > In this JIRA, the marked-for-deletion snapshots are hided so that they become > inaccessible > to users. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15482) Ordered snapshot deletion: hide the deleted snapshots from users
Tsz-wo Sze created HDFS-15482: - Summary: Ordered snapshot deletion: hide the deleted snapshots from users Key: HDFS-15482 URL: https://issues.apache.org/jira/browse/HDFS-15482 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tsz-wo Sze In HDFS-15480, the behavior of deleting the non-earliest snapshots is changed to marking them as deleted in XAttr but not actually deleting them. The users are still able to access the these snapshots as usual. In this JIRA, the marked-for-deletion snapshots are hided so that they become inaccessible to users. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15481) Ordered snapshot deletion: garbage collect deleted snapshots
Tsz-wo Sze created HDFS-15481: - Summary: Ordered snapshot deletion: garbage collect deleted snapshots Key: HDFS-15481 URL: https://issues.apache.org/jira/browse/HDFS-15481 Project: Hadoop HDFS Issue Type: Sub-task Components: snapshots Reporter: Tsz-wo Sze When the earliest snapshot is actually deleted, if the subsequent snapshots are already marked as deleted, the subsequent snapshots can be also actually removed from the file system. In this JIRA, we implement a mechanism to garbage collect these snapshots. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15479) Ordered snapshot deletion: make it a configurable feature
[ https://issues.apache.org/jira/browse/HDFS-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz-wo Sze updated HDFS-15479: -- Summary: Ordered snapshot deletion: make it a configurable feature (was: Add a conf for enforcing ordered snapshot deletion) > Ordered snapshot deletion: make it a configurable feature > - > > Key: HDFS-15479 > URL: https://issues.apache.org/jira/browse/HDFS-15479 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > > Ordered snapshot deletion is a configurable feature. In this JIRA, a conf is > added. > When the feature is enabled, only the earliest snapshot can be deleted. For > deleting the non-earliest snapshots, the behavior is temporarily changed to > throwing an exception in this JIRA. In HDFS-15480, the behavior of deleting > the non-earliest snapshots will be changed to marking them as deleted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15479) Add a conf for enforcing ordered snapshot deletion
[ https://issues.apache.org/jira/browse/HDFS-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz-wo Sze updated HDFS-15479: -- Description: Ordered snapshot deletion is a configurable feature. In this JIRA, a conf is added. When the feature is enabled, only the earliest snapshot can be deleted. For deleting the non-earliest snapshots, the behavior is temporarily changed to throwing an exception in this JIRA. In HDFS-15480, the behavior of deleting the non-earliest snapshots will be changed to marking them as deleted. was: Ordered snapshot deletion is a configurable feature. In this JIRA, a conf is added. When the feature is enabled, only the earliest snapshot can be deleted. For deleting the non-earliest snapshots, the behavior is temporarily changed to throwing an exception in this JIRA. In the next JIRA, the behavior of deleting the non-earliest snapshots will be changed to marking them as deleted. > Add a conf for enforcing ordered snapshot deletion > -- > > Key: HDFS-15479 > URL: https://issues.apache.org/jira/browse/HDFS-15479 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > > Ordered snapshot deletion is a configurable feature. In this JIRA, a conf is > added. > When the feature is enabled, only the earliest snapshot can be deleted. For > deleting the non-earliest snapshots, the behavior is temporarily changed to > throwing an exception in this JIRA. In HDFS-15480, the behavior of deleting > the non-earliest snapshots will be changed to marking them as deleted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15480) Ordered snapshot deletion: record snapshot deletion in XAttr
Tsz-wo Sze created HDFS-15480: - Summary: Ordered snapshot deletion: record snapshot deletion in XAttr Key: HDFS-15480 URL: https://issues.apache.org/jira/browse/HDFS-15480 Project: Hadoop HDFS Issue Type: Sub-task Components: snapshots Reporter: Tsz-wo Sze In this JIRA, the behavior of deleting the non-earliest snapshots will be changed to marking them as deleted in XAttr but not actually deleting them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15479) Add a conf for enforcing ordered snapshot deletion
Tsz-wo Sze created HDFS-15479: - Summary: Add a conf for enforcing ordered snapshot deletion Key: HDFS-15479 URL: https://issues.apache.org/jira/browse/HDFS-15479 Project: Hadoop HDFS Issue Type: Sub-task Components: snapshots Reporter: Tsz-wo Sze Assignee: Tsz-wo Sze Ordered snapshot deletion is a configurable feature. In this JIRA, a conf is added. When the feature is enabled, only the earliest snapshot can be deleted. For deleting the non-earliest snapshots, the behavior is temporarily changed to throwing an exception in this JIRA. In the next JIRA, the behavior of deleting the non-earliest snapshots will be changed to marking them as deleted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15477) Enforce ordered snapshot deletion
[ https://issues.apache.org/jira/browse/HDFS-15477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz-wo Sze updated HDFS-15477: -- Description: Snapshot deletion has caused a few bugs earlier such as HDFS-13101 and HDFS-15313. In this JIRA, we propose enforcing ordered snapshot deletion -- only the earliest snapshot is actually deleted from the file system. The other snapshots are only marked as deleted. They will be actually deleted from the file system until all the earlier snapshots are deleted. The reason of enforcing ordered snapshot deletion is based on the observation that the logic of deleting the earliest snapshot is much simpler since the prior snapshot does not exist. All the previous bugs are caused by removing inodes from the prior snapshots. One drawback of ordered snapshot deletion is that the resources are not yet released for deleting the non-earliest snapshots since they are only marked as deleted but not actually deleted. was: Snapshot deletion has caused a few bugs earlier such as HDFS-13101 and HDFS-15313. In this JIRA, we propose enforcing ordered snapshot deletion -- only the earliest snapshot is actually deleted from the file system. The other snapshots are only marked as deleted. They will be actually deleted from the file system until all the earlier snapshots are deleted. The reason of enforcing ordered snapshot deletion is based on the observation that the logic of deleting the earliest snapshot is much simpler since the prior snapshot does not exist. All the previous bugs are caused by removing inodes from the prior snapshots. One drawback of ordered snapshot deletion is that the non-earliest snapshots are only marked as deleted but not actually deleted. The resources are not yet released. > Enforce ordered snapshot deletion > - > > Key: HDFS-15477 > URL: https://issues.apache.org/jira/browse/HDFS-15477 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > > Snapshot deletion has caused a few bugs earlier such as HDFS-13101 and > HDFS-15313. In this JIRA, we propose enforcing ordered snapshot deletion -- > only the earliest snapshot is actually deleted from the file system. The > other snapshots are only marked as deleted. They will be actually deleted > from the file system until all the earlier snapshots are deleted. > The reason of enforcing ordered snapshot deletion is based on the observation > that the logic of deleting the earliest snapshot is much simpler since the > prior snapshot does not exist. All the previous bugs are caused by removing > inodes from the prior snapshots. > One drawback of ordered snapshot deletion is that the resources are not yet > released for deleting the non-earliest snapshots since they are only marked > as deleted but not actually deleted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15477) Enforce ordered snapshot deletion
[ https://issues.apache.org/jira/browse/HDFS-15477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz-wo Sze updated HDFS-15477: -- Description: Snapshot deletion has caused a few bugs earlier such as HDFS-13101 and HDFS-15313. In this JIRA, we propose enforcing ordered snapshot deletion -- only the earliest snapshot is actually deleted from the file system. The other snapshots are only marked as deleted. They will be actually deleted from the file system until all the earlier snapshots are deleted. The reason of enforcing ordered snapshot deletion is based on the observation that the logic of deleting the earliest snapshot is much simpler since the prior snapshot does not exist. All the previous bugs are caused by removing inodes from the prior snapshot. One drawback of ordered snapshot deletion is that the resources are not yet released for deleting the non-earliest snapshots since they are only marked as deleted but not actually deleted. was: Snapshot deletion has caused a few bugs earlier such as HDFS-13101 and HDFS-15313. In this JIRA, we propose enforcing ordered snapshot deletion -- only the earliest snapshot is actually deleted from the file system. The other snapshots are only marked as deleted. They will be actually deleted from the file system until all the earlier snapshots are deleted. The reason of enforcing ordered snapshot deletion is based on the observation that the logic of deleting the earliest snapshot is much simpler since the prior snapshot does not exist. All the previous bugs are caused by removing inodes from the prior snapshots. One drawback of ordered snapshot deletion is that the resources are not yet released for deleting the non-earliest snapshots since they are only marked as deleted but not actually deleted. > Enforce ordered snapshot deletion > - > > Key: HDFS-15477 > URL: https://issues.apache.org/jira/browse/HDFS-15477 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > > Snapshot deletion has caused a few bugs earlier such as HDFS-13101 and > HDFS-15313. In this JIRA, we propose enforcing ordered snapshot deletion -- > only the earliest snapshot is actually deleted from the file system. The > other snapshots are only marked as deleted. They will be actually deleted > from the file system until all the earlier snapshots are deleted. > The reason of enforcing ordered snapshot deletion is based on the observation > that the logic of deleting the earliest snapshot is much simpler since the > prior snapshot does not exist. All the previous bugs are caused by removing > inodes from the prior snapshot. > One drawback of ordered snapshot deletion is that the resources are not yet > released for deleting the non-earliest snapshots since they are only marked > as deleted but not actually deleted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15098) Add SM4 encryption method for HDFS
[
https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17160633#comment-17160633
]
Hadoop QA commented on HDFS-15098:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
55s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} prototool {color} | {color:blue} 0m
0s{color} | {color:blue} prototool was not available. {color} |
| {color:blue}0{color} | {color:blue} markdownlint {color} | {color:blue} 0m
0s{color} | {color:blue} markdownlint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
24m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
35s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
36s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
36s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 2
extant findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
33s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 4 extant
findbugs warnings. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 20m 32s{color} |
{color:red} root generated 34 new + 128 unchanged - 34 fixed = 162 total (was
162) {color} |
| {color:green}+1{color} | {color:green} golang {color} | {color:green} 20m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 20m 32s{color}
| {color:red} root generated 4 new + 1948 unchanged - 0 fixed = 1952 total (was
1948) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 7s{color} | {color:orange} root: The patch generated 7 new + 229 unchanged
- 8 fixed = 236 total (was 237) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 7 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 48s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green}
[jira] [Work started] (HDFS-15476) Make AsyncStream class' executor_ member private
[ https://issues.apache.org/jira/browse/HDFS-15476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-15476 started by Suraj Naik. - > Make AsyncStream class' executor_ member private > > > Key: HDFS-15476 > URL: https://issues.apache.org/jira/browse/HDFS-15476 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build, libhdfs++ >Reporter: Suraj Naik >Assignee: Suraj Naik >Priority: Minor > Fix For: 3.4.0 > > > As part of [HDFS-15385|https://issues.apache.org/jira/browse/HDFS-15385] the > boost library was upgraded. > The AsyncStream class has a getter function which returns the executor. > Keeping the executor member public makes the getter function's role > pointless. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15476) Make AsyncStream class' executor_ member private
[ https://issues.apache.org/jira/browse/HDFS-15476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Naik reassigned HDFS-15476: - Assignee: Suraj Naik > Make AsyncStream class' executor_ member private > > > Key: HDFS-15476 > URL: https://issues.apache.org/jira/browse/HDFS-15476 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build, libhdfs++ >Reporter: Suraj Naik >Assignee: Suraj Naik >Priority: Minor > Fix For: 3.4.0 > > > As part of [HDFS-15385|https://issues.apache.org/jira/browse/HDFS-15385] the > boost library was upgraded. > The AsyncStream class has a getter function which returns the executor. > Keeping the executor member public makes the getter function's role > pointless. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15476) Make AsyncStream class' executor_ member private
[ https://issues.apache.org/jira/browse/HDFS-15476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17160606#comment-17160606 ] Surendra Singh Lilhore commented on HDFS-15476: --- Added [~Suraj Naik] in HDFS contributor list. > Make AsyncStream class' executor_ member private > > > Key: HDFS-15476 > URL: https://issues.apache.org/jira/browse/HDFS-15476 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build, libhdfs++ >Reporter: Suraj Naik >Priority: Minor > Fix For: 3.4.0 > > > As part of [HDFS-15385|https://issues.apache.org/jira/browse/HDFS-15385] the > boost library was upgraded. > The AsyncStream class has a getter function which returns the executor. > Keeping the executor member public makes the getter function's role > pointless. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15478) When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs.
Uma Maheswara Rao G created HDFS-15478: -- Summary: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. Key: HDFS-15478 URL: https://issues.apache.org/jira/browse/HDFS-15478 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 3.2.1 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G On empty mount tables detection, we will automatically assign fallback with the same initialized uri fs. Currently we are using given uri for creating target fs. When creating target fs, we use Chrooted fs where it will set the path from uri as base directory. So, this can make path wrong in the case of fs initialized with path. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
