[jira] [Updated] (HDFS-1968) Enhance TestWriteRead to support File Append and Position Read
[ https://issues.apache.org/jira/browse/HDFS-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CW Chung updated HDFS-1968: --- Attachment: TestWriteRead.patch Only one patch is allowed. The formating part was taken care by HDFS-2024. This patch contains material changes only. > Enhance TestWriteRead to support File Append and Position Read > --- > > Key: HDFS-1968 > URL: https://issues.apache.org/jira/browse/HDFS-1968 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: CW Chung >Assignee: CW Chung >Priority: Minor > Attachments: TestWriteRead-1-Format.patch, > TestWriteRead-2-Append.patch, TestWriteRead.patch, TestWriteRead.patch, > TestWriteRead.patch, TestWriteRead.patch > > > Desirable to enhance TestWriteRead to support command line options to do: > (1) File Append > (2) Position Read (currently supporting sequential read). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2024) Eclipse format HDFS Junit test hdfs/TestWriteRead.java
[ https://issues.apache.org/jira/browse/HDFS-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CW Chung updated HDFS-2024: --- Attachment: TestWriteRead-2024.patch This patch contain just formatting change. No material change here. > Eclipse format HDFS Junit test hdfs/TestWriteRead.java > --- > > Key: HDFS-2024 > URL: https://issues.apache.org/jira/browse/HDFS-2024 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Reporter: CW Chung >Assignee: CW Chung >Priority: Trivial > Attachments: TestWriteRead-2024.patch > > > Eclipse format the file src/test/../hdfs/TestWriteRead.java. This is in > preparation of HDFS-1968. > So the patch should have only formatting changes such as white space. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2024) Eclipse format HDFS Junit test hdfs/TestWriteRead.java
Eclipse format HDFS Junit test hdfs/TestWriteRead.java --- Key: HDFS-2024 URL: https://issues.apache.org/jira/browse/HDFS-2024 Project: Hadoop HDFS Issue Type: Improvement Components: test Reporter: CW Chung Assignee: CW Chung Priority: Trivial Eclipse format the file src/test/../hdfs/TestWriteRead.java. This is in preparation of HDFS-1968. So the patch should have only formatting changes such as white space. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1968) Enhance TestWriteRead to support File Append and Position Read
[ https://issues.apache.org/jira/browse/HDFS-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CW Chung updated HDFS-1968: --- Attachment: TestWriteRead-2-Append.patch This is part 2 of the patch. This is the material change portion of code from svn. It is a diff from part 1 (which consist of only the formatting change). So to reveal / commit, apply the 2 patches in this order: a. Apply patch TestWriteRead-1-Format.patch to get to the version with better formating b. Apply patch TestWriteRead-2-Append.patch to get to the version with material changes. (Sorry for the formatting trouble. Next time I'll either do eclipse format right from the start, or never do it! ) > Enhance TestWriteRead to support File Append and Position Read > --- > > Key: HDFS-1968 > URL: https://issues.apache.org/jira/browse/HDFS-1968 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: CW Chung >Assignee: CW Chung >Priority: Minor > Attachments: TestWriteRead-1-Format.patch, > TestWriteRead-2-Append.patch, TestWriteRead.patch, TestWriteRead.patch, > TestWriteRead.patch > > > Desirable to enhance TestWriteRead to support command line options to do: > (1) File Append > (2) Position Read (currently supporting sequential read). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1968) Enhance TestWriteRead to support File Append and Position Read
[ https://issues.apache.org/jira/browse/HDFS-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CW Chung updated HDFS-1968: --- Attachment: TestWriteRead-1-Format.patch I have patch available to address Comment # 1 by John. To address the comment of Cos, I am dividing the patch into two parts: a. A patch (this file) to just re-format of the existing svn copy (basically eclipse format + some manual fix up). Since there is no material code change here, the hope is to get this committed quickly, then step b can be started. b. I then generate a patch on a (basically diff my latest version with the new formatted version). This patch would require real review. > Enhance TestWriteRead to support File Append and Position Read > --- > > Key: HDFS-1968 > URL: https://issues.apache.org/jira/browse/HDFS-1968 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: CW Chung >Assignee: CW Chung >Priority: Minor > Attachments: TestWriteRead-1-Format.patch, TestWriteRead.patch, > TestWriteRead.patch, TestWriteRead.patch > > > Desirable to enhance TestWriteRead to support command line options to do: > (1) File Append > (2) Position Read (currently supporting sequential read). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2021) HDFS Junit test TestWriteRead failed with inconsistent visible length of a file
HDFS Junit test TestWriteRead failed with inconsistent visible length of a file Key: HDFS-2021 URL: https://issues.apache.org/jira/browse/HDFS-2021 Project: Hadoop HDFS Issue Type: Bug Environment: Linux RHEL5 Reporter: CW Chung Assignee: John George Priority: Minor The junit test failed when iterates a number of times with larger chunk size on Linux. Once a while, the visible number of bytes seen by a reader is slightly less than what was supposed to be. When run with the following parameter, it failed more often on Linux ( as reported by John George) than my Mac: private static final int WR_NTIMES = 300; private static final int WR_CHUNK_SIZE = 1; Adding more debugging output to the source, this is a sample of the output: Caused by: java.io.IOException: readData mismatch in byte read: expected=277 ; got 2765312 at org.apache.hadoop.hdfs.TestWriteRead.readData(TestWriteRead.java:141) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1968) Enhance TestWriteRead to support File Append and Position Read
[ https://issues.apache.org/jira/browse/HDFS-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CW Chung updated HDFS-1968: --- Attachment: TestWriteRead.patch Improvement over the last patch: 1. Throw exception immediately on error 2. writeData routine will throw exception if actual data written is not the same amount as request 3. Print out more debugging info > Enhance TestWriteRead to support File Append and Position Read > --- > > Key: HDFS-1968 > URL: https://issues.apache.org/jira/browse/HDFS-1968 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: CW Chung >Assignee: CW Chung >Priority: Minor > Attachments: TestWriteRead.patch, TestWriteRead.patch, > TestWriteRead.patch > > > Desirable to enhance TestWriteRead to support command line options to do: > (1) File Append > (2) Position Read (currently supporting sequential read). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1968) Enhance TestWriteRead to support File Append and Position Read
[ https://issues.apache.org/jira/browse/HDFS-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CW Chung updated HDFS-1968: --- Attachment: TestWriteRead.patch The patch support the following additional command line options: [-append | -truncate]: if file already exists, truncate to 0 length before writing to the file. Default is append to file. [-usePosRead | - useSeqRead]: use position read to read the file, or the default sequential read. Note: Sorry for the large number of difference. The Eclipse Format did a number of cosmetic changes in the spacing. > Enhance TestWriteRead to support File Append and Position Read > --- > > Key: HDFS-1968 > URL: https://issues.apache.org/jira/browse/HDFS-1968 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: CW Chung >Assignee: CW Chung >Priority: Minor > Attachments: TestWriteRead.patch, TestWriteRead.patch > > > Desirable to enhance TestWriteRead to support command line options to do: > (1) File Append > (2) Position Read (currently supporting sequential read). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1968) Enhance TestWriteRead to support File Append and Position Read
[ https://issues.apache.org/jira/browse/HDFS-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CW Chung updated HDFS-1968: --- Attachment: TestWriteRead.patch This time with the proper granting license to ASF. > Enhance TestWriteRead to support File Append and Position Read > --- > > Key: HDFS-1968 > URL: https://issues.apache.org/jira/browse/HDFS-1968 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: CW Chung >Assignee: CW Chung >Priority: Minor > Attachments: TestWriteRead.patch, TestWriteRead.patch > > > Desirable to enhance TestWriteRead to support command line options to do: > (1) File Append > (2) Position Read (currently supporting sequential read). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1968) Enhance TestWriteRead to support File Append and Position Read
Enhance TestWriteRead to support File Append and Position Read --- Key: HDFS-1968 URL: https://issues.apache.org/jira/browse/HDFS-1968 Project: Hadoop HDFS Issue Type: Test Components: test Affects Versions: 0.23.0 Reporter: CW Chung Assignee: CW Chung Priority: Minor Desirable to enhance TestWriteRead to support command line options to do: (1) File Append (2) Position Read (currently supporting sequential read). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1877) Create a functional test for file read/write
[ https://issues.apache.org/jira/browse/HDFS-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CW Chung updated HDFS-1877: --- Attachment: TestWriteRead.patch Grant license to Apache. Otherwise, this version is the same as the one submitted 2 hours ago. > Create a functional test for file read/write > > > Key: HDFS-1877 > URL: https://issues.apache.org/jira/browse/HDFS-1877 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: CW Chung >Assignee: CW Chung >Priority: Minor > Attachments: TestWriteRead.java, TestWriteRead.patch, > TestWriteRead.patch, TestWriteRead.patch > > > It would be a great to have a tool, running on a real grid, to perform > function test (and stress tests to certain extent) for the file operations. > The tool would be written in Java and makes HDFS API calls to read, write, > append, hflush hadoop files. The tool would be usable standalone, or as a > building block for other regression or stress test suites (written in shell, > perl, python, etc). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1877) Create a functional test for file read/write
[ https://issues.apache.org/jira/browse/HDFS-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CW Chung updated HDFS-1877: --- Attachment: TestWriteRead.patch Implement suggestions by Nicholas. Thanks! > Create a functional test for file read/write > > > Key: HDFS-1877 > URL: https://issues.apache.org/jira/browse/HDFS-1877 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.22.0 >Reporter: CW Chung >Priority: Minor > Attachments: TestWriteRead.java, TestWriteRead.patch, > TestWriteRead.patch > > > It would be a great to have a tool, running on a real grid, to perform > function test (and stress tests to certain extent) for the file operations. > The tool would be written in Java and makes HDFS API calls to read, write, > append, hflush hadoop files. The tool would be usable standalone, or as a > building block for other regression or stress test suites (written in shell, > perl, python, etc). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1877) Create a functional test for file read/write
[ https://issues.apache.org/jira/browse/HDFS-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CW Chung updated HDFS-1877: --- Attachment: TestWriteRead.patch > Create a functional test for file read/write > > > Key: HDFS-1877 > URL: https://issues.apache.org/jira/browse/HDFS-1877 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.22.0 >Reporter: CW Chung >Priority: Minor > Attachments: TestWriteRead.java, TestWriteRead.patch > > > It would be a great to have a tool, running on a real grid, to perform > function test (and stress tests to certain extent) for the file operations. > The tool would be written in Java and makes HDFS API calls to read, write, > append, hflush hadoop files. The tool would be usable standalone, or as a > building block for other regression or stress test suites (written in shell, > perl, python, etc). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1877) Create a functional test for file read/write
[ https://issues.apache.org/jira/browse/HDFS-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CW Chung updated HDFS-1877: --- Status: Patch Available (was: Open) Thanks for the suggestion. Here is another patch with the following improvements: 1. Use svn diff format 2. System.out.println changed to LOG 3. Use sequential read rather than position read. Junit Test will therefore pass. 4. Removed unused import, declarations 5. Test routine begin with name Test 6. Add parsing of command line option > Create a functional test for file read/write > > > Key: HDFS-1877 > URL: https://issues.apache.org/jira/browse/HDFS-1877 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.22.0 >Reporter: CW Chung >Priority: Minor > Attachments: TestWriteRead.java > > > It would be a great to have a tool, running on a real grid, to perform > function test (and stress tests to certain extent) for the file operations. > The tool would be written in Java and makes HDFS API calls to read, write, > append, hflush hadoop files. The tool would be usable standalone, or as a > building block for other regression or stress test suites (written in shell, > perl, python, etc). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1877) Create a functional test for file read/write
[ https://issues.apache.org/jira/browse/HDFS-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CW Chung updated HDFS-1877: --- Attachment: TestWriteRead.java > Create a functional test for file read/write > > > Key: HDFS-1877 > URL: https://issues.apache.org/jira/browse/HDFS-1877 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.22.0 >Reporter: CW Chung >Priority: Minor > Attachments: TestWriteRead.java > > > It would be a great to have a tool, running on a real grid, to perform > function test (and stress tests to certain extent) for the file operations. > The tool would be written in Java and makes HDFS API calls to read, write, > append, hflush hadoop files. The tool would be usable standalone, or as a > building block for other regression or stress test suites (written in shell, > perl, python, etc). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1877) Create a functional test for file read/write
[ https://issues.apache.org/jira/browse/HDFS-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CW Chung updated HDFS-1877: --- Status: Open (was: Patch Available) > Create a functional test for file read/write > > > Key: HDFS-1877 > URL: https://issues.apache.org/jira/browse/HDFS-1877 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.22.0 >Reporter: CW Chung >Priority: Minor > > It would be a great to have a tool, running on a real grid, to perform > function test (and stress tests to certain extent) for the file operations. > The tool would be written in Java and makes HDFS API calls to read, write, > append, hflush hadoop files. The tool would be usable standalone, or as a > building block for other regression or stress test suites (written in shell, > perl, python, etc). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1877) Create a functional test for file read/write
[ https://issues.apache.org/jira/browse/HDFS-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CW Chung updated HDFS-1877: --- Status: Patch Available (was: Open) This test can be run both as a JUnit test and as a Standlone test in a real-cluster. This test will generate an exception (BlockMissingException). > Create a functional test for file read/write > > > Key: HDFS-1877 > URL: https://issues.apache.org/jira/browse/HDFS-1877 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.22.0 >Reporter: CW Chung >Priority: Minor > > It would be a great to have a tool, running on a real grid, to perform > function test (and stress tests to certain extent) for the file operations. > The tool would be written in Java and makes HDFS API calls to read, write, > append, hflush hadoop files. The tool would be usable standalone, or as a > building block for other regression or stress test suites (written in shell, > perl, python, etc). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1907) BlockMissingException upon concurrent read and write: reader was doing file position read while writer is doing write without hflush
[ https://issues.apache.org/jira/browse/HDFS-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030986#comment-13030986 ] CW Chung commented on HDFS-1907: Stack Trace: = 11/05/08 19:05:48 WARN hdfs.DFSClient: Failed to connect to /xxx.xx.xx.xxx:1004 for file /tmp/N/909NF for block BP-1632719171-xxx.xx.xx.xxx-1303748685682:blk_-8940328094159486414_3653:java.io.IOException: Got error for OP_READ_BLOCK, self=/xxx.xx.xx.xxx:36405, remote=/xxx.xx.xx.xxx:1004, for file /tmp/N/909NF, for pool BP-1632719171-xxx.xx.xx.xxx-1303748685682 block -8940328094159486414_3653 at org.apache.hadoop.hdfs.BlockReader.newBlockReader(BlockReader.java:398) at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:631) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:704) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:51) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:192) Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1632719171-xxx.xx.xx.xxx-1303748685682:blk_-8940328094159486414_3653 file=/tmp/N/909NF at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:570) at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:618) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:704) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:51) > BlockMissingException upon concurrent read and write: reader was doing file > position read while writer is doing write without hflush > > > Key: HDFS-1907 > URL: https://issues.apache.org/jira/browse/HDFS-1907 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Affects Versions: 0.23.0 > Environment: Run on a real cluster. Using the latest 0.23 build. >Reporter: CW Chung > > BlockMissingException is thrown under this test scenario: > Two different processes doing concurrent file r/w: one read and the other > write on the same file > - writer keep doing file write > - reader doing position file read from beginning of the file to the visible > end of file, repeatedly > The reader is basically doing: > byteRead = in.read(currentPosition, buffer, 0, byteToReadThisRound); > where CurrentPostion=0, buffer is a byte array buffer, byteToReadThisRound = > 1024*1; > Usually it does not fail right away. I have to read, close file, re-open the > same file a few times to create the problem. I'll pose a test program to > repro this problem after I've cleaned up a bit my current test program. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1907) BlockMissingException upon concurrent read and write: reader was doing file position read while writer is doing write without hflush
[ https://issues.apache.org/jira/browse/HDFS-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CW Chung updated HDFS-1907: --- Description: BlockMissingException is thrown under this test scenario: Two different processes doing concurrent file r/w: one read and the other write on the same file - writer keep doing file write - reader doing position file read from beginning of the file to the visible end of file, repeatedly The reader is basically doing: byteRead = in.read(currentPosition, buffer, 0, byteToReadThisRound); where CurrentPostion=0, buffer is a byte array buffer, byteToReadThisRound = 1024*1; Usually it does not fail right away. I have to read, close file, re-open the same file a few times to create the problem. I'll pose a test program to repro this problem after I've cleaned up a bit my current test program. was: BlockMissingException is thrown under this test scenario: - two different processes doing concurrent file r/w: one read and the other write on the same file - writer keep doing file write - reader doing position file read from beginning of the file to the visible end of file, repeatedly The call is basically: byteRead = in.read(currentPosition, buffer, 0, byteToReadThisRound); where CurrentPostion=0, buffer is a byte array buffer, byteToReadThisRound = 1024*1; Usually it does not fail right away. I have to read, close file, re-open the same file a few times to create the problem. I'll pose a test program to repro this problem after I've cleaned up a bit my current test program. Error msg: = 11/05/08 19:05:48 WARN hdfs.DFSClient: Failed to connect to /xxx.xx.xx.xxx:1004 for file /tmp/N/909NF for block BP-1632719171-xxx.xx.xx.xxx-1303748685682:blk_-8940328094159486414_3653:java.io.IOException: Got error for OP_READ_BLOCK, self=/xxx.xx.xx.xxx:36405, remote=/xxx.xx.xx.xxx:1004, for file /tmp/N/909NF, for pool BP-1632719171-xxx.xx.xx.xxx-1303748685682 block -8940328094159486414_3653 at org.apache.hadoop.hdfs.BlockReader.newBlockReader(BlockReader.java:398) at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:631) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:704) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:51) at TAppend.readUntilVisibleEnd(TAppend.java:441) at TAppend.readUntilEnd(TAppend.java:474) at TAppend.testReadOnly(TAppend.java:956) at TAppend.main(TAppend.java:1215) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:192) # Caught Exception in testReadOnly while reading. java.io.IOException: Exception caught in readUntilVisibleEnd: Reader currentOffset = 0 ; totalByteRead =0 ; latest byteRead = 0 ; visibleLen= 67824640 ; byteLeftToRead = 67824640 ; bufferLen = 1024 ; chunkNumber= 0 ; input pos = 0 ; byteToReadThisRound = 1024 ; Filename = /tmp/N/909NF, ReadParam - CurrentPostion=0, offset=0, size=1024 at TAppend.readUntilVisibleEnd(TAppend.java:457) at TAppend.readUntilEnd(TAppend.java:474) at TAppend.testReadOnly(TAppend.java:956) at TAppend.main(TAppend.java:1215) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:192) Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1632719171-xxx.xx.xx.xxx-1303748685682:blk_-8940328094159486414_3653 file=/tmp/N/909NF at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:570) at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:618) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:704) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:51) at TAppend.readUntilVisibleEnd(TAppend.java:441) ... 8 more > BlockMissingException upon concurrent read and write: reader was doing file > position read while writer is doing write without hflush > > > Key: HDFS-1907 > URL: https://issues.apache.org/jira/browse/HDFS-1907 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Affects Versions: 0.23.0 > Environment: Ru
[jira] [Updated] (HDFS-1907) BlockMissingException upon concurrent read and write: reader was doing file position read while writer is doing write without hflush
[ https://issues.apache.org/jira/browse/HDFS-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CW Chung updated HDFS-1907: --- Description: BlockMissingException is thrown under this test scenario: - two different processes doing concurrent file r/w: one read and the other write on the same file - writer keep doing file write - reader doing position file read from beginning of the file to the visible end of file, repeatedly The call is basically: byteRead = in.read(currentPosition, buffer, 0, byteToReadThisRound); where CurrentPostion=0, buffer is a byte array buffer, byteToReadThisRound = 1024*1; Usually it does not fail right away. I have to read, close file, re-open the same file a few times to create the problem. I'll pose a test program to repro this problem after I've cleaned up a bit my current test program. Error msg: = 11/05/08 19:05:48 WARN hdfs.DFSClient: Failed to connect to /xxx.xx.xx.xxx:1004 for file /tmp/N/909NF for block BP-1632719171-xxx.xx.xx.xxx-1303748685682:blk_-8940328094159486414_3653:java.io.IOException: Got error for OP_READ_BLOCK, self=/xxx.xx.xx.xxx:36405, remote=/xxx.xx.xx.xxx:1004, for file /tmp/N/909NF, for pool BP-1632719171-xxx.xx.xx.xxx-1303748685682 block -8940328094159486414_3653 at org.apache.hadoop.hdfs.BlockReader.newBlockReader(BlockReader.java:398) at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:631) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:704) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:51) at TAppend.readUntilVisibleEnd(TAppend.java:441) at TAppend.readUntilEnd(TAppend.java:474) at TAppend.testReadOnly(TAppend.java:956) at TAppend.main(TAppend.java:1215) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:192) # Caught Exception in testReadOnly while reading. java.io.IOException: Exception caught in readUntilVisibleEnd: Reader currentOffset = 0 ; totalByteRead =0 ; latest byteRead = 0 ; visibleLen= 67824640 ; byteLeftToRead = 67824640 ; bufferLen = 1024 ; chunkNumber= 0 ; input pos = 0 ; byteToReadThisRound = 1024 ; Filename = /tmp/N/909NF, ReadParam - CurrentPostion=0, offset=0, size=1024 at TAppend.readUntilVisibleEnd(TAppend.java:457) at TAppend.readUntilEnd(TAppend.java:474) at TAppend.testReadOnly(TAppend.java:956) at TAppend.main(TAppend.java:1215) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:192) Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1632719171-xxx.xx.xx.xxx-1303748685682:blk_-8940328094159486414_3653 file=/tmp/N/909NF at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:570) at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:618) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:704) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:51) at TAppend.readUntilVisibleEnd(TAppend.java:441) ... 8 more was: BlockMissingException is thrown under this test scenario: - two different processes doing concurrent file r/w: one read and the other write on the same file - writer keep doing file write - reader doing position file read from beginning of the file to the visible end of file, repeatedly The call is basically: byteRead = in.read(currentPosition, buffer, 0, byteToReadThisRound); where CurrentPostion=0, buffer is a byte array buffer, byteToReadThisRound = 1024*1; Usually it does not fail right away. I have to read, close file, re-open the same file a few times to create the problem. I'll pose a test program to repro this problem after I've cleaned up a bit my current test program. Error msg: = 11/05/08 19:05:48 WARN hdfs.DFSClient: Failed to connect to /98.137.98.116:1004 for file /tmp/N/909NF for block BP-1632719171-98.137.98.114-1303748685682:blk_-8940328094159486414_3653:java.io.IOException: Got error for OP_READ_BLOCK, self=/98.137.98.125:36405, remote=/98.137.98.116:1004, for file /tmp/N/909NF, for pool BP-1632719171-98.137.98.114-1303748685682 block -8940328094159486414_3653 at org.apache.hadoop.hdfs.BlockReader.newBlockReader(BlockReader.java:398) at org.apache.hadoo
[jira] [Created] (HDFS-1907) BlockMissingException upon concurrent read and write: reader was doing file position read while writer is doing write without hflush
BlockMissingException upon concurrent read and write: reader was doing file position read while writer is doing write without hflush Key: HDFS-1907 URL: https://issues.apache.org/jira/browse/HDFS-1907 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.23.0 Environment: Run on a real cluster $ hadoop version Hadoop 0.22.0.1105090202 Subversion git://hadoopre5.corp.sk1.yahoo.com/home/y/var/builds/thread2/workspace/Cloud-HadoopCommon-0.22-Secondary -r 3c23e43f9e262e7843e4287436429fad3224b0f7 Compiled by hadoopqa on Mon May 9 02:13:09 PDT 2011 >From source with checksum 90b5fc469fd7a1fa0ba22db893423fed Reporter: CW Chung BlockMissingException is thrown under this test scenario: - two different processes doing concurrent file r/w: one read and the other write on the same file - writer keep doing file write - reader doing position file read from beginning of the file to the visible end of file, repeatedly The call is basically: byteRead = in.read(currentPosition, buffer, 0, byteToReadThisRound); where CurrentPostion=0, buffer is a byte array buffer, byteToReadThisRound = 1024*1; Usually it does not fail right away. I have to read, close file, re-open the same file a few times to create the problem. I'll pose a test program to repro this problem after I've cleaned up a bit my current test program. Error msg: = 11/05/08 19:05:48 WARN hdfs.DFSClient: Failed to connect to /98.137.98.116:1004 for file /tmp/N/909NF for block BP-1632719171-98.137.98.114-1303748685682:blk_-8940328094159486414_3653:java.io.IOException: Got error for OP_READ_BLOCK, self=/98.137.98.125:36405, remote=/98.137.98.116:1004, for file /tmp/N/909NF, for pool BP-1632719171-98.137.98.114-1303748685682 block -8940328094159486414_3653 at org.apache.hadoop.hdfs.BlockReader.newBlockReader(BlockReader.java:398) at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:631) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:704) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:51) at TAppend.readUntilVisibleEnd(TAppend.java:441) at TAppend.readUntilEnd(TAppend.java:474) at TAppend.testReadOnly(TAppend.java:956) at TAppend.main(TAppend.java:1215) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:192) # Caught Exception in testReadOnly while reading. java.io.IOException: Exception caught in readUntilVisibleEnd: Reader currentOffset = 0 ; totalByteRead =0 ; latest byteRead = 0 ; visibleLen= 67824640 ; byteLeftToRead = 67824640 ; bufferLen = 1024 ; chunkNumber= 0 ; input pos = 0 ; byteToReadThisRound = 1024 ; Filename = /tmp/N/909NF, ReadParam - CurrentPostion=0, offset=0, size=1024 at TAppend.readUntilVisibleEnd(TAppend.java:457) at TAppend.readUntilEnd(TAppend.java:474) at TAppend.testReadOnly(TAppend.java:956) at TAppend.main(TAppend.java:1215) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:192) Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1632719171-98.137.98.114-1303748685682:blk_-8940328094159486414_3653 file=/tmp/N/909NF at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:570) at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:618) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:704) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:51) at TAppend.readUntilVisibleEnd(TAppend.java:441) ... 8 more -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1877) Create a functional test for file read/write
[ https://issues.apache.org/jira/browse/HDFS-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029015#comment-13029015 ] CW Chung commented on HDFS-1877: Yes, the first approach is closer to what I have in mind. I should be able to upload a patch for comment pretty soon. Thanks for pointing out HDFS-1762. The TestHDFSCLI looks to be an interesting exercise to make it run in a real cluster. > Create a functional test for file read/write > > > Key: HDFS-1877 > URL: https://issues.apache.org/jira/browse/HDFS-1877 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.22.0 >Reporter: CW Chung >Priority: Minor > > It would be a great to have a tool, running on a real grid, to perform > function test (and stress tests to certain extent) for the file operations. > The tool would be written in Java and makes HDFS API calls to read, write, > append, hflush hadoop files. The tool would be usable standalone, or as a > building block for other regression or stress test suites (written in shell, > perl, python, etc). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1877) Create a functional test for file read/write
[ https://issues.apache.org/jira/browse/HDFS-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028981#comment-13028981 ] CW Chung commented on HDFS-1877: I would like to add to the hadoop-test.jar such that there is minimal hassle in using it. The source likely would be in hadoop-hdfs src/test/hdfs/org/apache/hadoop/hdfs. > Create a functional test for file read/write > > > Key: HDFS-1877 > URL: https://issues.apache.org/jira/browse/HDFS-1877 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.22.0 >Reporter: CW Chung >Priority: Minor > > It would be a great to have a tool, running on a real grid, to perform > function test (and stress tests to certain extent) for the file operations. > The tool would be written in Java and makes HDFS API calls to read, write, > append, hflush hadoop files. The tool would be usable standalone, or as a > building block for other regression or stress test suites (written in shell, > perl, python, etc). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1877) Create a functional test for file read/write
[ https://issues.apache.org/jira/browse/HDFS-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CW Chung updated HDFS-1877: --- Summary: Create a functional test for file read/write (was: Create a functional test tool for file read/write) > Create a functional test for file read/write > > > Key: HDFS-1877 > URL: https://issues.apache.org/jira/browse/HDFS-1877 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.22.0 >Reporter: CW Chung >Priority: Minor > > It would be a great to have a tool, running on a real grid, to perform > function test (and stress tests to certain extent) for the file operations. > The tool would be written in Java and makes HDFS API calls to read, write, > append, hflush hadoop files. The tool would be usable standalone, or as a > building block for other regression or stress test suites (written in shell, > perl, python, etc). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1877) Create a functional test tool for file read/write
[ https://issues.apache.org/jira/browse/HDFS-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028557#comment-13028557 ] CW Chung commented on HDFS-1877: Yes, this is a system test for HDFS. It is meant to be a small, quick-to-implement test that can be run on a test or production cluster, without any special library or change in cluster configuration. Although it can be used as a building block for system regression tests, it is nowhere near the scope or complexity of Herriot or S-LIVE/DFSIO. So let's call it a test rather than a tool. > Create a functional test tool for file read/write > - > > Key: HDFS-1877 > URL: https://issues.apache.org/jira/browse/HDFS-1877 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.22.0 >Reporter: CW Chung >Priority: Minor > > It would be a great to have a tool, running on a real grid, to perform > function test (and stress tests to certain extent) for the file operations. > The tool would be written in Java and makes HDFS API calls to read, write, > append, hflush hadoop files. The tool would be usable standalone, or as a > building block for other regression or stress test suites (written in shell, > perl, python, etc). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1877) Create a functional test tool for file read/write
[ https://issues.apache.org/jira/browse/HDFS-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028477#comment-13028477 ] CW Chung commented on HDFS-1877: Good question. The differences are: 1. This tool would be run as a vanilla client, not as map/reduce tasks. So you can test HDFS file operations without depending on another subsystem(map/reduce). 2. The focus of the tool is not in the performance, but in the verification of functional correctness. > Create a functional test tool for file read/write > - > > Key: HDFS-1877 > URL: https://issues.apache.org/jira/browse/HDFS-1877 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.22.0 >Reporter: CW Chung >Priority: Minor > > It would be a great to have a tool, running on a real grid, to perform > function test (and stress tests to certain extent) for the file operations. > The tool would be written in Java and makes HDFS API calls to read, write, > append, hflush hadoop files. The tool would be usable standalone, or as a > building block for other regression or stress test suites (written in shell, > perl, python, etc). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1877) Create a functional test tool for file read/write
Create a functional test tool for file read/write - Key: HDFS-1877 URL: https://issues.apache.org/jira/browse/HDFS-1877 Project: Hadoop HDFS Issue Type: Test Components: test Affects Versions: 0.22.0 Reporter: CW Chung Priority: Minor It would be a great to have a tool, running on a real grid, to perform function test (and stress tests to certain extent) for the file operations. The tool would be written in Java and makes HDFS API calls to read, write, append, hflush hadoop files. The tool would be usable standalone, or as a building block for other regression or stress test suites (written in shell, perl, python, etc). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
