[
https://issues.apache.org/jira/browse/MAPREDUCE-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900654#action_12900654
]
Hong Tang commented on MAPREDUCE-2023:
--------------------------------------
To confirm that DFS indeed may return less than requested bytes even before
reaching the end, I wrote a test program (attached), and the output of a sample
run looks as follows:
{noformat}
hadoop dfs -ls /user/gridperf/gridmix3/part-m-00332/segment-0
Found 1 items
-rw-rw-rw- 3 gridperf hdfs 1073741824 2010-08-20 08:22
/user/gridperf/gridmix3/part-m-00332/segment-0
hadoop org.apache.hadoop.fs.TestFsRead
/user/gridperf/gridmix3/part-m-00332/segment-0 1000000
10995954 bytes read
21199983 bytes read
32106261 bytes read
42209617 bytes read
52456131 bytes read
63551911 bytes read
73836262 bytes read
84369397 bytes read
95182878 bytes read
105047397 bytes read
115740295 bytes read
126323360 bytes read
137166764 bytes read
147066000 bytes read
157744477 bytes read
168319334 bytes read
178856592 bytes read
188884554 bytes read
199324045 bytes read
209995098 bytes read
220916802 bytes read
231218738 bytes read
241772291 bytes read
251883835 bytes read
262306687 bytes read
Fail to read a full buffer before reaching the end: pos=267640189,
expected=994623, actual=795267.
272862612 bytes read
283737254 bytes read
293851212 bytes read
304525446 bytes read
314766024 bytes read
325604342 bytes read
335604768 bytes read
346475397 bytes read
357311830 bytes read
367574920 bytes read
377834612 bytes read
388029682 bytes read
398728223 bytes read
408966064 bytes read
419626247 bytes read
430260987 bytes read
440440647 bytes read
451030835 bytes read
461808645 bytes read
471996795 bytes read
482529325 bytes read
493106417 bytes read
503960340 bytes read
514155195 bytes read
524460261 bytes read
534955349 bytes read
Fail to read a full buffer before reaching the end: pos=536250423,
expected=999458, actual=620489.
545734170 bytes read
556326582 bytes read
567046173 bytes read
577480068 bytes read
587338410 bytes read
598115745 bytes read
608759717 bytes read
619418792 bytes read
629597629 bytes read
639906390 bytes read
650264871 bytes read
661414262 bytes read
671205472 bytes read
681856772 bytes read
692394138 bytes read
702803762 bytes read
713182701 bytes read
723720128 bytes read
734531251 bytes read
745188960 bytes read
755814801 bytes read
765670009 bytes read
776047213 bytes read
786592324 bytes read
797786600 bytes read
Fail to read a full buffer before reaching the end: pos=804788073,
expected=613320, actual=518295.
808158276 bytes read
818373817 bytes read
828549794 bytes read
838915719 bytes read
850189376 bytes read
860102547 bytes read
870902116 bytes read
881206170 bytes read
891441081 bytes read
902119052 bytes read
912394977 bytes read
923010497 bytes read
933330792 bytes read
944216276 bytes read
954226049 bytes read
965371734 bytes read
975663038 bytes read
986215681 bytes read
996274088 bytes read
1006954729 bytes read
1017375248 bytes read
1027801749 bytes read
1038384467 bytes read
1049383853 bytes read
1059662742 bytes read
1070106760 bytes read
{noformat}
> TestDFSIO read test may not read specified bytes.
> -------------------------------------------------
>
> Key: MAPREDUCE-2023
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks
> Reporter: Hong Tang
>
> TestDFSIO's read test may read less bytes than specified when reading large
> files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
