[jira] [Commented] (HADOOP-16255) ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum
[ https://issues.apache.org/jira/browse/HADOOP-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128724#comment-17128724 ] Jungtaek Lim commented on HADOOP-16255: --- Thanks [~kihwal] to deal with this in Hadoop 2.x! > ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum > -- > > Key: HADOOP-16255 > URL: https://issues.apache.org/jira/browse/HADOOP-16255 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.8.5, 3.1.2 >Reporter: Steve Loughran >Assignee: Jungtaek Lim >Priority: Major > Fix For: 2.8.6, 3.2.1, 2.9.3, 2.10.1, 3.1.5 > > Attachments: HADOOP-16255-branch-2-001.patch > > > ChecksumFS doesn't override FilterFS rename/3, so doesn't rename the checksum > with the file. > As a result, if a file is renamed over an existing file using rename(src, > dest, OVERWRITE) the renamed file will be considered to have an invalid > checksum -the old one is picked up instead. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16255) ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum
[ https://issues.apache.org/jira/browse/HADOOP-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128690#comment-17128690 ] Kihwal Lee commented on HADOOP-16255: - Cherry-picked the change to branch-3.1. Committed the branch-2 patch to branch-2.10 and also cherry-picked it to branch-2.9 and branch-2.8. Branch 2.9 and 2.8 are EOL except for security fixes, but some of us are still tracking the branches. > ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum > -- > > Key: HADOOP-16255 > URL: https://issues.apache.org/jira/browse/HADOOP-16255 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.8.5, 3.1.2 >Reporter: Steve Loughran >Assignee: Jungtaek Lim >Priority: Major > Fix For: 2.8.6, 3.2.1, 2.10.1, 3.1.5 > > Attachments: HADOOP-16255-branch-2-001.patch > > > ChecksumFS doesn't override FilterFS rename/3, so doesn't rename the checksum > with the file. > As a result, if a file is renamed over an existing file using rename(src, > dest, OVERWRITE) the renamed file will be considered to have an invalid > checksum -the old one is picked up instead. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16255) ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum
[ https://issues.apache.org/jira/browse/HADOOP-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128685#comment-17128685 ] Kihwal Lee commented on HADOOP-16255: - This needs to be in all active branches. > ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum > -- > > Key: HADOOP-16255 > URL: https://issues.apache.org/jira/browse/HADOOP-16255 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.8.5, 3.1.2 >Reporter: Steve Loughran >Assignee: Jungtaek Lim >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-16255-branch-2-001.patch > > > ChecksumFS doesn't override FilterFS rename/3, so doesn't rename the checksum > with the file. > As a result, if a file is renamed over an existing file using rename(src, > dest, OVERWRITE) the renamed file will be considered to have an invalid > checksum -the old one is picked up instead. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16255) ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum
[ https://issues.apache.org/jira/browse/HADOOP-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927199#comment-16927199 ] Jungtaek Lim commented on HADOOP-16255: --- [~ste...@apache.org] Just attached the patch file for branch-2 as guided. Thanks! > ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum > -- > > Key: HADOOP-16255 > URL: https://issues.apache.org/jira/browse/HADOOP-16255 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.8.5, 3.1.2 >Reporter: Steve Loughran >Assignee: Jungtaek Lim >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-16255-branch-2-001.patch > > > ChecksumFS doesn't override FilterFS rename/3, so doesn't rename the checksum > with the file. > As a result, if a file is renamed over an existing file using rename(src, > dest, OVERWRITE) the renamed file will be considered to have an invalid > checksum -the old one is picked up instead. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16255) ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum
[ https://issues.apache.org/jira/browse/HADOOP-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924588#comment-16924588 ] Hudson commented on HADOOP-16255: - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17247 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17247/]) HADOOP-16255. Add ChecksumFs.rename(path, path, boolean) (stevel: rev bb0b922a71cba9ceaf00588e9f3e3b2a3c2e3eab) * (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestChecksumFs.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFs.java > ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum > -- > > Key: HADOOP-16255 > URL: https://issues.apache.org/jira/browse/HADOOP-16255 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.8.5, 3.1.2 >Reporter: Steve Loughran >Assignee: Jungtaek Lim >Priority: Major > > ChecksumFS doesn't override FilterFS rename/3, so doesn't rename the checksum > with the file. > As a result, if a file is renamed over an existing file using rename(src, > dest, OVERWRITE) the renamed file will be considered to have an invalid > checksum -the old one is picked up instead. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16255) ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum
[ https://issues.apache.org/jira/browse/HADOOP-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921593#comment-16921593 ] Steve Loughran commented on HADOOP-16255: - 2.x clients should all be able to talk to 3.x clusters, if not -complain. The biggest troublespot is invariably live upgrades of HDFS clusters. Spark does not have to worry about that directly > ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum > -- > > Key: HADOOP-16255 > URL: https://issues.apache.org/jira/browse/HADOOP-16255 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.8.5, 3.1.2 >Reporter: Steve Loughran >Assignee: Jungtaek Lim >Priority: Major > > ChecksumFS doesn't override FilterFS rename/3, so doesn't rename the checksum > with the file. > As a result, if a file is renamed over an existing file using rename(src, > dest, OVERWRITE) the renamed file will be considered to have an invalid > checksum -the old one is picked up instead. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16255) ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum
[ https://issues.apache.org/jira/browse/HADOOP-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920017#comment-16920017 ] Jungtaek Lim commented on HADOOP-16255: --- [https://github.com/apache/hadoop/pull/1388] > ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum > -- > > Key: HADOOP-16255 > URL: https://issues.apache.org/jira/browse/HADOOP-16255 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.8.5, 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > ChecksumFS doesn't override FilterFS rename/3, so doesn't rename the checksum > with the file. > As a result, if a file is renamed over an existing file using rename(src, > dest, OVERWRITE) the renamed file will be considered to have an invalid > checksum -the old one is picked up instead. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16255) ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum
[ https://issues.apache.org/jira/browse/HADOOP-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919986#comment-16919986 ] Jungtaek Lim commented on HADOOP-16255: --- Ah OK. Good to know that 2.7.x is EOL. Spark 3.0 is planning to support both 2.7.4 and 3.2.0, and upgrading Hadoop version could affect many things like transitive dependency, compatibility between old Hadoop server (does Hadoop client guarantee backward compatibility?), etc. I haven't contributed Hadoop project so may need to take time to do it. I'll see how I can contribute it, but you can also go ahead if I'm taking too much time to do it. > ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum > -- > > Key: HADOOP-16255 > URL: https://issues.apache.org/jira/browse/HADOOP-16255 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.8.5, 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > ChecksumFS doesn't override FilterFS rename/3, so doesn't rename the checksum > with the file. > As a result, if a file is renamed over an existing file using rename(src, > dest, OVERWRITE) the renamed file will be considered to have an invalid > checksum -the old one is picked up instead. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16255) ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum
[ https://issues.apache.org/jira/browse/HADOOP-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914524#comment-16914524 ] Steve Loughran commented on HADOOP-16255: - I see your issue. # It should just be matter of overriding the rename/3 call and having it move the CRC tool; this is good to preserve the checksum too. # patches with tests welcome... # we're never going to do another 2.7.x release so saying it is affected isn't going to make anyone care. I've tagged 2.8.5 as affected too # now. can you upgrade spark's dependencies? please? > ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum > -- > > Key: HADOOP-16255 > URL: https://issues.apache.org/jira/browse/HADOOP-16255 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.8.5, 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > ChecksumFS doesn't override FilterFS rename/3, so doesn't rename the checksum > with the file. > As a result, if a file is renamed over an existing file using rename(src, > dest, OVERWRITE) the renamed file will be considered to have an invalid > checksum -the old one is picked up instead. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16255) ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum
[ https://issues.apache.org/jira/browse/HADOOP-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913832#comment-16913832 ] Jungtaek Lim commented on HADOOP-16255: --- [~ste...@apache.org] Spark also hits this issue https://issues.apache.org/jira/browse/SPARK-28025 though the concern is a bit different. Once "crc for src" is not renamed, it's a leak as we have no opaque way to delete this. We'll just delete the crc file as an workaround but definitely looking forward to remove the hack. Btw, the Hadoop version Spark is using is 2.7.4, but it seems to be a long standing issue through various version lines (maybe all 2.x versions, or even earlier?). What would be the preferred way to do this? Filing new issue describing Spark's case, or could you please add 2.7.4 to the affected version/s here? > ChecksumFS.Make FileSystem.rename(path, path, options) doesn't rename checksum > -- > > Key: HADOOP-16255 > URL: https://issues.apache.org/jira/browse/HADOOP-16255 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > ChecksumFS doesn't override FilterFS rename/3, so doesn't rename the checksum > with the file. > As a result, if a file is renamed over an existing file using rename(src, > dest, OVERWRITE) the renamed file will be considered to have an invalid > checksum -the old one is picked up instead. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
