[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14962297#comment-14962297
]
Steve Loughran commented on HADOOP-11244:
-
this got duplicated & fixed by HADOOP-12268. Sorry Jay -you did get this patch
in first, but we weren't keeping atop the patches enough.
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Noah Watkins
>Assignee: jay vyas
> Labels: BB2015-05-TBR
> Fix For: 2.8.0
>
> Attachments: HADOOP-11244.patch, HADOOP-11244.patch
>
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14354334#comment-14354334
]
Hadoop QA commented on HADOOP-11244:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678241/HADOOP-11244.patch
against trunk revision c3003eb.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.fs.contract.localfs.TestLocalFSContractAppend
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5902//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5902//console
This message is automatically generated.
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Noah Watkins
>Assignee: jay vyas
> Attachments: HADOOP-11244.patch, HADOOP-11244.patch
>
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14257104#comment-14257104
]
jay vyas commented on HADOOP-11244:
---
okay, so delete *testRenameFileBeingAppended* ?
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Noah Watkins
>Assignee: jay vyas
> Fix For: 2.7.0
>
> Attachments: HADOOP-11244.patch, HADOOP-11244.patch
>
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14229957#comment-14229957
]
Steve Loughran commented on HADOOP-11244:
-
OK, let's just cull the tests. It doesn't really prove anything either way, we
just need to say "no guarantees of anything on rename during append"
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Noah Watkins
>Assignee: jay vyas
> Fix For: 2.7.0
>
> Attachments: HADOOP-11244.patch, HADOOP-11244.patch
>
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14196139#comment-14196139
]
jay vyas commented on HADOOP-11244:
---
hi [~ste...@apache.org] id propose
- first, we remove this test from the code.
- then, fix HADOOP-11251, make all the tests run.
-in the meanwhile, we can leisurely decide how to test
testRenameFileBeingAppended test .
Right now, given the complexity of the issue and time constraints, *i think its
best to remove dead code so HCFS tests work perfectly again*. let me know
thoughts on this (or maybe another plan forward if you have one)?
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Noah Watkins
>Assignee: jay vyas
> Fix For: 2.6.0
>
> Attachments: HADOOP-11244.patch, HADOOP-11244.patch
>
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191821#comment-14191821
]
jay vyas commented on HADOOP-11244:
---
*summary so far*
- AbstractContractAppendTest is (1) not being run (2) not capable of passing
- next step : Make it run in LocalFS tests as part of hadoops standard CI (high
priority)
- Also, make this specific task (testRenameFileBeingAppended) either : pass
*or* just delete the method entirely.
just let us know what needs to be done next ! thanks steve !
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Noah Watkins
>Assignee: jay vyas
> Fix For: 2.6.0
>
> Attachments: HADOOP-11244.patch, HADOOP-11244.patch
>
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191816#comment-14191816
]
jay vyas commented on HADOOP-11244:
---
Hi [~ste...@apache.org] : Looks like this code, in your old repo, has the test:
{noformat}
git clone https://github.com/steveloughran/hadoop-trunk.git
git checkout remotes/origin/old/HADOOP-9361-filesystem-contract
vim
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractAppend.java
{noformat}
I don't see it anywhere else though ?
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Noah Watkins
>Assignee: jay vyas
> Fix For: 2.6.0
>
> Attachments: HADOOP-11244.patch, HADOOP-11244.patch
>
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191692#comment-14191692
]
Steve Loughran commented on HADOOP-11244:
-
not thread safety: visibility of changes to the FS to your code.
Examples
# file deleted while you have the stream open for reading
# file appended to during reads
# full R/W files: changes to the contents of a file during a read
# multiple writers appending: are the appends atomic vs interleaved
that's what I was trying to say. Now, if that wasn't clear, that could go into
a patch
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Noah Watkins
>Assignee: jay vyas
> Fix For: 2.6.0
>
> Attachments: HADOOP-11244.patch, HADOOP-11244.patch
>
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190805#comment-14190805
]
Noah Watkins commented on HADOOP-11244:
---
I read "no guarantees on concurrent operations" to refer to the thread safety
of an implementation of the file system object, but the scenario in this test
can occur for a single thread. Is that the intended meanin gof the spec?
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Noah Watkins
>Assignee: jay vyas
> Fix For: 2.6.0
>
> Attachments: HADOOP-11244.patch, HADOOP-11244.patch
>
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190799#comment-14190799
]
jay vyas commented on HADOOP-11244:
---
okay, so then this test should be deleted - . but lets keep the part of
the patch which turns *on* the {{fs.contract.supports-append}} to prevent this
from happening again.
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Noah Watkins
>Assignee: jay vyas
> Fix For: 2.6.0
>
> Attachments: HADOOP-11244.patch, HADOOP-11244.patch
>
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190762#comment-14190762
]
Steve Loughran commented on HADOOP-11244:
-
filesystem spec explicitly says "no guarantees on concurrent operations"
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Noah Watkins
>Assignee: jay vyas
> Fix For: 2.6.0
>
> Attachments: HADOOP-11244.patch, HADOOP-11244.patch
>
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190498#comment-14190498
]
Hadoop QA commented on HADOOP-11244:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678241/HADOOP-11244.patch
against trunk revision 73e626a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.fs.contract.localfs.TestLocalFSContractAppend
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4990//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4990//console
This message is automatically generated.
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Noah Watkins
>Assignee: jay vyas
> Fix For: 2.6.0
>
> Attachments: HADOOP-11244.patch, HADOOP-11244.patch
>
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190358#comment-14190358
]
jay vyas commented on HADOOP-11244:
---
i agree have created HADOOP-11251, let me look into that and ill make it a
blocker
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Noah Watkins
>Assignee: jay vyas
> Fix For: 2.6.0
>
> Attachments: HADOOP-11244.patch
>
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190351#comment-14190351
]
Steve Loughran commented on HADOOP-11244:
-
# need to understand why the raw local tests aren't running first, because they
should -and this problem should have surfaced. if they aren't running, we don't
catch regressions
# what happens to all the other filesystems with this test. I'd expect the
object stores to fail because they cache the destination on output stream
creation, and only write on close().
if we are getting inconsistent outcomes, this will have to become one of those
fs switch things.
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Noah Watkins
>Assignee: jay vyas
> Fix For: 2.6.0
>
> Attachments: HADOOP-11244.patch
>
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190222#comment-14190222
]
Hadoop QA commented on HADOOP-11244:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678174/HADOOP-11244.patch
against trunk revision 179cab8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4989//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4989//console
This message is automatically generated.
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Noah Watkins
>Assignee: jay vyas
> Fix For: 2.6.0
>
> Attachments: HADOOP-11244.patch
>
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190113#comment-14190113
]
jay vyas commented on HADOOP-11244:
---
One more note : this is easy to test {{mvn test
-Dtest=org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractAppend}} for
noah/huamin if you want to confirm that it works. .
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Noah Watkins
>Assignee: jay vyas
> Attachments: HADOOP-11244.patch
>
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190099#comment-14190099
]
jay vyas commented on HADOOP-11244:
---
[~ste...@apache.org] if this is okay can we push this through ? Then i will
work on HADOOP-11251 (do our dillegence to confirm that other FS contract
tests are being run). thanks
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Noah Watkins
> Attachments: HADOOP-11244.patch
>
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190057#comment-14190057
]
Huamin Chen commented on HADOOP-11244:
--
+1 LGTM
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Noah Watkins
> Attachments: HADOOP-11244.patch
>
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14189950#comment-14189950
]
jay vyas commented on HADOOP-11244:
---
Thanks @stevel we will take it from here and get a patch up today or tomorrow.
For others reading this thread for now this test can be disabled using the
above XML attribute...
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Noah Watkins
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14189857#comment-14189857
]
Steve Loughran commented on HADOOP-11244:
-
yes, something seems awry, though why it ever passed now misses me.
the test is actually intended to stress something that proved pretty variable
across systems: what happens when you rename a file that is actively being
written to, with some expectation that things may happen.
the logic was meant to be
{code}
outputstream.write
Path renamed = new Path(testPath, "renamed");
rename(target, renamed); <- need rename here
outputStream.close();
{code}
There's two outcomes I'd expect
# data ends up on new path
# data ends up on old path
with the code in there you will automatically get outcome #2, but the test is
for #1. Now I'm confused.
I do not have any free time to look at this right now. I have pushed up to
github the branch with the full history:
[[https://github.com/steveloughran/hadoop-trunk/tree/old/HADOOP-9361-filesystem-contract]]
if something got dropped it should be in the history ... though the file got
renamed near the end so it's not so easy to use the github history to track down
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Noah Watkins
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188901#comment-14188901
]
jay vyas commented on HADOOP-11244:
---
yup, something looks funy
{noformat}
---
Running org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractAppend
Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.823 sec <<<
FAILURE! - in org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractAppend
testRenameFileBeingAppended(org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractAppend)
Time elapsed: 0.464 sec <<< ERROR!
java.io.FileNotFoundException: renamed destination file does not exist: not
found
file:/home/jay/Development/hadoop-common/hadoop-common-project/hadoop-common/target/test/data/test/renamed
in file:/home/jay/Development/hadoop-common/hadoo
p-common-project/hadoop-common/target/test/data/test
at
org.apache.hadoop.fs.contract.ContractTestUtils.assertPathExists(ContractTestUtils.java:678)
at
org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertPathExists(AbstractFSContractTestBase.java:279)
at
org.apache.hadoop.fs.contract.AbstractContractAppendTest.testRenameFileBeingAppended(AbstractContractAppendTest.java:121)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Results :
Tests in error:
TestRawlocalContractAppend>AbstractContractAppendTest.testRenameFileBeingAppended:121->AbstractFSContractTestBase.assertPathExists:279
ยป FileNotFound
{noformat}
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Noah Watkins
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188833#comment-14188833
]
jay vyas commented on HADOOP-11244:
---
Another clue Seems that these tests are being skipped in the hadoop build when
i run "mvn test -Dtest=org.apache.hadoop.fs.contract.rawlocal.*" looking
further now. My original assumption was that all these tests ran, so the error
must be on your end :) . but clearly, that might not be the case.
{noformat}
Running org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractOpen
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.707 sec - in
org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractOpen
Running org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractDelete
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.694 sec - in
org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractDelete
Running org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractAppend
Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 0.673 sec - in
org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractAppend
Running org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractMkdir
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.634 sec - in
org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractMkdir
Running org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractSeek
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.974 sec - in
org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractSeek
Running
org.apache.hadoop.fs.contract.rawlocal.TestRawLocalContractUnderlyingFileBehavior
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.165 sec - in
org.apache.hadoop.fs.contract.rawlocal.TestRawLocalContractUnderlyingFileBehavior
Running org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractCreate
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.682 sec - in
org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractCreate
Running org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractRename
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.681 sec - in
org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractRename
{noformat}
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Noah Watkins
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188805#comment-14188805
]
jay vyas commented on HADOOP-11244:
---
[~nwatkins] [~chenh]
looking at the code again, i guess your write (pun intended) - there is nothing
referencing the renamed object. ill run the RawLocalFS tests now and see how
this is even passing
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Noah Watkins
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188790#comment-14188790
]
Noah Watkins commented on HADOOP-11244:
---
Hmm, I think then I may be reading this code incorrectly.
Here we start writing to "target"
{noformat}
touch(getFileSystem(), target);
assertPathExists("original file does not exist", target);
byte[] dataset = dataset(256, 'a', 'z');
FSDataOutputStream outputStream = getFileSystem().append(target);
outputStream.write(dataset);
{noformat}
Create the path object for "renamed". I'm assuming this has no side effects.
{noformat}
Path renamed = new Path(testPath, "renamed");
{noformat}
Close the "target" stream, list the directory, and assert "renamed" file
exists.
{noformat}
outputStream.close();
String listing = ls(testPath);
assertPathExists("renamed destination file does not exist", renamed);
{noformat}
I don't see anything in the code that would result in "renamed" being created.
Am I getting the semantics of `new Path` wrong? Should that have side effects
on the last opened output stream?
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Noah Watkins
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188791#comment-14188791
]
Huamin Chen commented on HADOOP-11244:
--
I concur with [~nwatkins]. The `renamed` path is not associated with any
stream. The right code to me is
outputstream.write
Path renamed = new Path(testPath, "renamed");
outputStream.close();
rename(target, renamed); <- need rename here
String listing = ls(testPath);
//expected: the stream goes to the file that was being renamed, not the
original path
assertPathExists("renamed destination file does not exist", renamed);
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Noah Watkins
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188785#comment-14188785
]
jay vyas commented on HADOOP-11244:
---
make this change in your HCFS XML file
{noformat}
fs.contract.supports-append
false
{noformat}
This will disable the test. but i think it will be necessary for example for
something like hbase. renaming is defined as part of the append test file
suite it looks like.
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Noah Watkins
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188783#comment-14188783
]
jay vyas commented on HADOOP-11244:
---
After looking closely, i think it might not be a bug. lets take a look:
{noformat}
outputstream.write
Path renamed = new Path(testPath, "renamed");
outputStream.close();
String listing = ls(testPath);
//expected: the stream goes to the file that was being renamed, not the
original path
assertPathExists("renamed destination file does not exist", renamed);
{noformat}
I think the issue here, then, is that the stream , when being closed, should
forward write out to the renamed path.
Probably if you don't see this behaaviour, then there is a bug in your
filesystem implementation, which is that it is writing eagerly.
You can disable this test in via the controller XML file. I can try to lookup
the exact xml tag (but the basic idea is here
https://wiki.apache.org/hadoop/HCFS/Progress) .
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Noah Watkins
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188777#comment-14188777
]
Noah Watkins commented on HADOOP-11244:
---
The way I'm reading the code for the test, it looks to me like it isn't
actually testing what is intended.
I'm looking at the code here
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractAppendTest.java#L108
During the test the path to the renamed file is created "Path renamed = new
Path(testPath, "renamed");", but then that path is never created (ostensibly by
renaming the file opened for writing). After that it is asserted that the
renamed file exists. "assertPathExists("renamed destination file does not
exist", renamed);"
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Noah Watkins
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11244) The HCFS contract test testRenameFileBeingAppended doesn't do a rename
[
https://issues.apache.org/jira/browse/HADOOP-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188756#comment-14188756
]
jay vyas commented on HADOOP-11244:
---
thanks for reporting this. are you saying the test fails indefinetly ? I am
looking into this one now.
> The HCFS contract test testRenameFileBeingAppended doesn't do a rename
> --
>
> Key: HADOOP-11244
> URL: https://issues.apache.org/jira/browse/HADOOP-11244
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Noah Watkins
>
> The test AbstractContractAppendTest::testRenameFileBeingAppended appears to
> assert the behavior of renaming a file opened for writing. However, the
> assertion "assertPathExists("renamed destination file does not exist",
> renamed);" fails because it appears that the file "renamed" is never created
> (ostensibly it should be the "target" file that has been renamed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
