[
https://issues.apache.org/jira/browse/HADOOP-13871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733471#comment-15733471
]
Steve Loughran commented on HADOOP-13871:
-----------------------------------------
Patch for trunk adds a new {{resetConnection()}} operation which aborts any
active underlying connection, with this test modified to explicitly detect
reads too slow and perform resets, failing if >8 resets happen during the run.
This detects and reacts to bandwidth problems, albeit with a hard coded
threshold of 128KB/s. If you are getting less than that, you shouldn't be
running that test. Except: after the abort the TCP connection will be
slow-starting up to full B/W, so on any-long haul link, that next read may also
underperform. It's probably safe to have a lower value.
It would be possible to track bandwidth per
Example
{code}
2016-12-08 21:29:45,965 [Thread-1] DEBUG s3a.S3AFileSystem
(S3AUtils.java:intOption(512)) - Value of fs.s3a.connection.maximum is 25
2016-12-08 21:29:45,967 [Thread-1] DEBUG s3a.S3AFileSystem
(S3AUtils.java:intOption(512)) - Value of fs.s3a.attempts.maximum is 20
2016-12-08 21:29:45,967 [Thread-1] DEBUG s3a.S3AFileSystem
(S3AUtils.java:intOption(512)) - Value of fs.s3a.connection.establish.timeout
is 5000
2016-12-08 21:29:45,968 [Thread-1] DEBUG s3a.S3AFileSystem
(S3AUtils.java:intOption(512)) - Value of fs.s3a.connection.timeout is 5000
2016-12-08 21:29:45,968 [Thread-1] DEBUG s3a.S3AFileSystem
(S3AUtils.java:intOption(512)) - Value of fs.s3a.socket.send.buffer is 65536
2016-12-08 21:29:45,968 [Thread-1] DEBUG s3a.S3AFileSystem
(S3AUtils.java:intOption(512)) - Value of fs.s3a.socket.recv.buffer is 32678
2016-12-08 21:29:45,970 [Thread-1] DEBUG s3a.S3AFileSystem
(S3ClientFactory.java:initUserAgent(187)) - Using User-Agent: Hadoop
2.8.0-SNAPSHOT
2016-12-08 21:29:46,186 [Thread-1] DEBUG s3a.S3AFileSystem
(S3AUtils.java:intOption(512)) - Value of fs.s3a.paging.maximum is 5000
2016-12-08 21:29:46,188 [Thread-1] DEBUG s3a.S3AFileSystem
(S3AUtils.java:longBytesOption(555)) - Value of fs.s3a.block.size is 33554432
2016-12-08 21:29:46,188 [Thread-1] DEBUG s3a.S3AFileSystem
(S3AUtils.java:longBytesOption(555)) - Value of fs.s3a.readahead.range is 65536
2016-12-08 21:29:46,190 [Thread-1] DEBUG s3a.S3AFileSystem
(S3AUtils.java:intOption(512)) - Value of fs.s3a.max.total.tasks is 10
2016-12-08 21:29:46,190 [Thread-1] DEBUG s3a.S3AFileSystem
(S3AUtils.java:longOption(533)) - Value of fs.s3a.threads.keepalivetime is 60
2016-12-08 21:29:49,070 DEBUG amazonaws.request
(AmazonHttpClient.java:logResponseRequestId(856)) - AWS Request ID:
1FEDCD17B6FBB2C8
2016-12-08 21:29:49,381 DEBUG - Bytes in read #1: 16347 , block bytes: 16347,
remaining in block: 1032229 duration=478865598 nS; ns/byte: 29293,
bandwidth=0.032556 MB/s
2016-12-08 21:29:49,382 DEBUG - Bytes in read #2: 630 , block bytes: 16977,
remaining in block: 1031599 duration=212331 nS; ns/byte: 337,
bandwidth=2.829614 MB/s
2016-12-08 21:29:49,531 DEBUG - Bytes in read #3: 16347 , block bytes: 33324,
remaining in block: 1015252 duration=149393770 nS; ns/byte: 9138,
bandwidth=0.104353 MB/s
2016-12-08 21:29:49,532 DEBUG - Bytes in read #4: 1061 , block bytes: 34385,
remaining in block: 1014191 duration=167998 nS; ns/byte: 158,
bandwidth=6.022979 MB/s
2016-12-08 21:29:49,688 DEBUG - Bytes in read #5: 16347 , block bytes: 50732,
remaining in block: 997844 duration=155828299 nS; ns/byte: 9532,
bandwidth=0.100044 MB/s
2016-12-08 21:29:49,688 DEBUG - Bytes in read #6: 1061 , block bytes: 51793,
remaining in block: 996783 duration=128858 nS; ns/byte: 121, bandwidth=7.852430
MB/s
2016-12-08 21:29:49,841 DEBUG - Bytes in read #7: 16347 , block bytes: 68140,
remaining in block: 980436 duration=152823750 nS; ns/byte: 9348,
bandwidth=0.102011 MB/s
2016-12-08 21:29:49,842 DEBUG - Bytes in read #8: 1061 , block bytes: 69201,
remaining in block: 979375 duration=280146 nS; ns/byte: 264, bandwidth=3.611861
MB/s
2016-12-08 21:29:50,149 DEBUG - Bytes in read #9: 16347 , block bytes: 85548,
remaining in block: 963028 duration=306567695 nS; ns/byte: 18753,
bandwidth=0.050852 MB/s
2016-12-08 21:29:50,149 DEBUG - Bytes in read #10: 1061 , block bytes: 86609,
remaining in block: 961967 duration=161057 nS; ns/byte: 151, bandwidth=6.282549
MB/s
2016-12-08 21:29:51,073 DEBUG - Bytes in read #11: 16347 , block bytes: 102956,
remaining in block: 945620 duration=923209136 nS; ns/byte: 56475,
bandwidth=0.016886 MB/s
2016-12-08 21:29:51,073 DEBUG - Bytes in read #12: 1061 , block bytes: 104017,
remaining in block: 944559 duration=172071 nS; ns/byte: 162, bandwidth=5.880412
MB/s
2016-12-08 21:29:51,690 DEBUG - Bytes in read #13: 16347 , block bytes: 120364,
remaining in block: 928212 duration=615842374 nS; ns/byte: 37673,
bandwidth=0.025314 MB/s
2016-12-08 21:29:51,843 DEBUG - Bytes in read #14: 1061 , block bytes: 121425,
remaining in block: 927151 duration=152502578 nS; ns/byte: 143734,
bandwidth=0.006635 MB/s
2016-12-08 21:29:52,816 DEBUG - Bytes in read #15: 16347 , block bytes: 137772,
remaining in block: 910804 duration=972690659 nS; ns/byte: 59502,
bandwidth=0.016027 MB/s
2016-12-08 21:29:52,816 DEBUG - Bytes in read #16: 1061 , block bytes: 138833,
remaining in block: 909743 duration=154987 nS; ns/byte: 146, bandwidth=6.528602
MB/s
2016-12-08 21:29:53,889 DEBUG - Bytes in read #17: 16347 , block bytes: 155180,
remaining in block: 893396 duration=1072600915 nS; ns/byte: 65614,
bandwidth=0.014534 MB/s
2016-12-08 21:29:53,991 DEBUG - Bytes in read #18: 1061 , block bytes: 156241,
remaining in block: 892335 duration=101460691 nS; ns/byte: 95627,
bandwidth=0.009973 MB/s
2016-12-08 21:29:55,165 DEBUG - Bytes in read #19: 16347 , block bytes: 172588,
remaining in block: 875988 duration=1173337630 nS; ns/byte: 71776,
bandwidth=0.013287 MB/s
2016-12-08 21:29:55,165 DEBUG - Bytes in read #20: 1061 , block bytes: 173649,
remaining in block: 874927 duration=116827 nS; ns/byte: 110, bandwidth=8.661084
MB/s
2016-12-08 21:29:55,628 DEBUG - Bytes in read #21: 16347 , block bytes: 189996,
remaining in block: 858580 duration=462231152 nS; ns/byte: 28276,
bandwidth=0.033727 MB/s
2016-12-08 21:29:55,935 DEBUG - Bytes in read #22: 1061 , block bytes: 191057,
remaining in block: 857519 duration=306790128 nS; ns/byte: 289151,
bandwidth=0.003298 MB/s
2016-12-08 21:29:56,785 DEBUG - Bytes in read #23: 16347 , block bytes: 207404,
remaining in block: 841172 duration=849333859 nS; ns/byte: 51956,
bandwidth=0.018355 MB/s
2016-12-08 21:29:56,786 DEBUG - Bytes in read #24: 1061 , block bytes: 208465,
remaining in block: 840111 duration=223996 nS; ns/byte: 211, bandwidth=4.517261
MB/s
2016-12-08 21:29:58,197 DEBUG - Bytes in read #25: 16347 , block bytes: 224812,
remaining in block: 823764 duration=1411150169 nS; ns/byte: 86324,
bandwidth=0.011048 MB/s
2016-12-08 21:29:58,350 DEBUG - Bytes in read #26: 1061 , block bytes: 225873,
remaining in block: 822703 duration=151618968 nS; ns/byte: 142901,
bandwidth=0.006674 MB/s
2016-12-08 21:29:59,275 DEBUG - Bytes in read #27: 16347 , block bytes: 242220,
remaining in block: 806356 duration=924832331 nS; ns/byte: 56575,
bandwidth=0.016857 MB/s
2016-12-08 21:29:59,275 DEBUG - Bytes in read #28: 1061 , block bytes: 243281,
remaining in block: 805295 duration=157331 nS; ns/byte: 148, bandwidth=6.431336
MB/s
2016-12-08 21:30:00,197 DEBUG - Bytes in read #29: 16347 , block bytes: 259628,
remaining in block: 788948 duration=919177546 nS; ns/byte: 56229,
bandwidth=0.016961 MB/s
2016-12-08 21:30:00,197 DEBUG - Bytes in read #30: 1061 , block bytes: 260689,
remaining in block: 787887 duration=159754 nS; ns/byte: 150, bandwidth=6.333791
MB/s
2016-12-08 21:30:01,116 DEBUG - Bytes in read #31: 16347 , block bytes: 277036,
remaining in block: 771540 duration=918850994 nS; ns/byte: 56209,
bandwidth=0.016967 MB/s
2016-12-08 21:30:01,117 DEBUG - Bytes in read #32: 1061 , block bytes: 278097,
remaining in block: 770479 duration=153206 nS; ns/byte: 144, bandwidth=6.604496
MB/s
2016-12-08 21:30:02,040 DEBUG - Bytes in read #33: 16347 , block bytes: 294444,
remaining in block: 754132 duration=923043728 nS; ns/byte: 56465,
bandwidth=0.016889 MB/s
2016-12-08 21:30:02,041 DEBUG - Bytes in read #34: 1061 , block bytes: 295505,
remaining in block: 753071 duration=152100 nS; ns/byte: 143, bandwidth=6.652521
MB/s
2016-12-08 21:30:03,119 DEBUG - Bytes in read #35: 16347 , block bytes: 311852,
remaining in block: 736724 duration=1077535517 nS; ns/byte: 65916,
bandwidth=0.014468 MB/s
2016-12-08 21:30:03,119 DEBUG - Bytes in read #36: 1061 , block bytes: 312913,
remaining in block: 735663 duration=114098 nS; ns/byte: 107, bandwidth=8.868240
MB/s
2016-12-08 21:30:04,140 DEBUG - Bytes in read #37: 16347 , block bytes: 329260,
remaining in block: 719316 duration=1020943631 nS; ns/byte: 62454,
bandwidth=0.015270 MB/s
2016-12-08 21:30:04,141 DEBUG - Bytes in read #38: 1061 , block bytes: 330321,
remaining in block: 718255 duration=154676 nS; ns/byte: 145, bandwidth=6.541729
MB/s
2016-12-08 21:30:05,063 DEBUG - Bytes in read #39: 16347 , block bytes: 346668,
remaining in block: 701908 duration=921944703 nS; ns/byte: 56398,
bandwidth=0.016910 MB/s
2016-12-08 21:30:06,027 DEBUG - Bytes in read #40: 1061 , block bytes: 347729,
remaining in block: 700847 duration=963349047 nS; ns/byte: 907963,
bandwidth=0.001050 MB/s
2016-12-08 21:30:07,488 DEBUG - Bytes in read #41: 16347 , block bytes: 364076,
remaining in block: 684500 duration=1460036513 nS; ns/byte: 89315,
bandwidth=0.010678 MB/s
2016-12-08 21:30:07,795 DEBUG - Bytes in read #42: 1061 , block bytes: 365137,
remaining in block: 683439 duration=306386850 nS; ns/byte: 288771,
bandwidth=0.003303 MB/s
2016-12-08 21:30:08,130 DEBUG - Bytes in read #43: 16347 , block bytes: 381484,
remaining in block: 667092 duration=335450550 nS; ns/byte: 20520,
bandwidth=0.046474 MB/s
2016-12-08 21:30:08,262 DEBUG - Bytes in read #44: 1061 , block bytes: 382545,
remaining in block: 666031 duration=130994948 nS; ns/byte: 123463,
bandwidth=0.007724 MB/s
2016-12-08 21:30:08,595 DEBUG - Bytes in read #45: 16347 , block bytes: 398892,
remaining in block: 649684 duration=333166901 nS; ns/byte: 20380,
bandwidth=0.046793 MB/s
2016-12-08 21:30:08,596 DEBUG - Bytes in read #46: 1061 , block bytes: 399953,
remaining in block: 648623 duration=131882 nS; ns/byte: 124, bandwidth=7.672377
MB/s
2016-12-08 21:30:09,211 DEBUG - Bytes in read #47: 16347 , block bytes: 416300,
remaining in block: 632276 duration=614798623 nS; ns/byte: 37609,
bandwidth=0.025357 MB/s
2016-12-08 21:30:09,211 DEBUG - Bytes in read #48: 1061 , block bytes: 417361,
remaining in block: 631215 duration=186043 nS; ns/byte: 175, bandwidth=5.438788
MB/s
2016-12-08 21:30:09,674 DEBUG - Bytes in read #49: 16347 , block bytes: 433708,
remaining in block: 614868 duration=462221834 nS; ns/byte: 28275,
bandwidth=0.033728 MB/s
2016-12-08 21:30:09,674 DEBUG - Bytes in read #50: 1061 , block bytes: 434769,
remaining in block: 613807 duration=124757 nS; ns/byte: 117, bandwidth=8.110555
MB/s
2016-12-08 21:30:09,984 DEBUG - Bytes in read #51: 16347 , block bytes: 451116,
remaining in block: 597460 duration=308999007 nS; ns/byte: 18902,
bandwidth=0.050452 MB/s
2016-12-08 21:30:09,984 DEBUG - Bytes in read #52: 1061 , block bytes: 452177,
remaining in block: 596399 duration=84490 nS; ns/byte: 79, bandwidth=11.975955
MB/s
2016-12-08 21:30:10,597 DEBUG - Bytes in read #53: 16347 , block bytes: 468524,
remaining in block: 580052 duration=612539269 nS; ns/byte: 37471,
bandwidth=0.025451 MB/s
2016-12-08 21:30:10,597 DEBUG - Bytes in read #54: 1061 , block bytes: 469585,
remaining in block: 578991 duration=147609 nS; ns/byte: 139, bandwidth=6.854924
MB/s
2016-12-08 21:30:11,060 DEBUG - Bytes in read #55: 16347 , block bytes: 485932,
remaining in block: 562644 duration=462436225 nS; ns/byte: 28288,
bandwidth=0.033712 MB/s
2016-12-08 21:30:11,060 DEBUG - Bytes in read #56: 1061 , block bytes: 486993,
remaining in block: 561583 duration=100576 nS; ns/byte: 94, bandwidth=10.060536
MB/s
2016-12-08 21:30:11,931 DEBUG - Bytes in read #57: 16347 , block bytes: 503340,
remaining in block: 545236 duration=870489198 nS; ns/byte: 53250,
bandwidth=0.017909 MB/s
2016-12-08 21:30:11,981 DEBUG - Bytes in read #58: 1061 , block bytes: 504401,
remaining in block: 544175 duration=49541837 nS; ns/byte: 46693,
bandwidth=0.020424 MB/s
2016-12-08 21:30:13,060 DEBUG - Bytes in read #59: 16347 , block bytes: 520748,
remaining in block: 527828 duration=1078170319 nS; ns/byte: 65955,
bandwidth=0.014459 MB/s
2016-12-08 21:30:13,060 DEBUG - Bytes in read #60: 1061 , block bytes: 521809,
remaining in block: 526767 duration=88905 nS; ns/byte: 83, bandwidth=11.381232
MB/s
2016-12-08 21:30:13,832 DEBUG - Bytes in read #61: 16347 , block bytes: 538156,
remaining in block: 510420 duration=771710163 nS; ns/byte: 47208,
bandwidth=0.020202 MB/s
2016-12-08 21:30:13,833 DEBUG - Bytes in read #62: 1061 , block bytes: 539217,
remaining in block: 509359 duration=195473 nS; ns/byte: 184, bandwidth=5.176410
MB/s
2016-12-08 21:30:14,985 DEBUG - Bytes in read #63: 16347 , block bytes: 555564,
remaining in block: 493012 duration=1152517210 nS; ns/byte: 70503,
bandwidth=0.013527 MB/s
2016-12-08 21:30:14,986 DEBUG - Bytes in read #64: 1061 , block bytes: 556625,
remaining in block: 491951 duration=137869 nS; ns/byte: 129, bandwidth=7.339202
MB/s
2016-12-08 21:30:15,756 DEBUG - Bytes in read #65: 16347 , block bytes: 572972,
remaining in block: 475604 duration=770143725 nS; ns/byte: 47112,
bandwidth=0.020243 MB/s
2016-12-08 21:30:15,757 DEBUG - Bytes in read #66: 1061 , block bytes: 574033,
remaining in block: 474543 duration=161316 nS; ns/byte: 152, bandwidth=6.272462
MB/s
2016-12-08 21:30:16,907 DEBUG - Bytes in read #67: 16347 , block bytes: 590380,
remaining in block: 458196 duration=1149954555 nS; ns/byte: 70346,
bandwidth=0.013557 MB/s
2016-12-08 21:30:16,908 DEBUG - Bytes in read #68: 1061 , block bytes: 591441,
remaining in block: 457135 duration=166415 nS; ns/byte: 156, bandwidth=6.080272
MB/s
2016-12-08 21:30:17,776 DEBUG - Bytes in read #69: 16347 , block bytes: 607788,
remaining in block: 440788 duration=867495557 nS; ns/byte: 53067,
bandwidth=0.017971 MB/s
2016-12-08 21:30:17,777 DEBUG - Bytes in read #70: 1061 , block bytes: 608849,
remaining in block: 439727 duration=938146 nS; ns/byte: 884, bandwidth=1.078562
MB/s
2016-12-08 21:30:18,948 DEBUG - Bytes in read #71: 16347 , block bytes: 625196,
remaining in block: 423380 duration=1170544837 nS; ns/byte: 71606,
bandwidth=0.013318 MB/s
2016-12-08 21:30:19,256 DEBUG - Bytes in read #72: 1061 , block bytes: 626257,
remaining in block: 422319 duration=307536130 nS; ns/byte: 289854,
bandwidth=0.003290 MB/s
2016-12-08 21:30:19,721 DEBUG - Bytes in read #73: 16347 , block bytes: 642604,
remaining in block: 405972 duration=464770187 nS; ns/byte: 28431,
bandwidth=0.033543 MB/s
2016-12-08 21:30:19,721 DEBUG - Bytes in read #74: 1061 , block bytes: 643665,
remaining in block: 404911 duration=145775 nS; ns/byte: 137, bandwidth=6.941166
MB/s
2016-12-08 21:30:20,030 DEBUG - Bytes in read #75: 16347 , block bytes: 660012,
remaining in block: 388564 duration=307973423 nS; ns/byte: 18839,
bandwidth=0.050620 MB/s
2016-12-08 21:30:20,338 DEBUG - Bytes in read #76: 1061 , block bytes: 661073,
remaining in block: 387503 duration=307555981 nS; ns/byte: 289873,
bandwidth=0.003290 MB/s
2016-12-08 21:30:20,798 DEBUG - Bytes in read #77: 16347 , block bytes: 677420,
remaining in block: 371156 duration=459891175 nS; ns/byte: 28133,
bandwidth=0.033899 MB/s
2016-12-08 21:30:20,951 DEBUG - Bytes in read #78: 1061 , block bytes: 678481,
remaining in block: 370095 duration=152066369 nS; ns/byte: 143323,
bandwidth=0.006654 MB/s
2016-12-08 21:30:21,414 DEBUG - Bytes in read #79: 16347 , block bytes: 694828,
remaining in block: 353748 duration=463288362 nS; ns/byte: 28340,
bandwidth=0.033650 MB/s
2016-12-08 21:30:21,565 DEBUG - Bytes in read #80: 1061 , block bytes: 695889,
remaining in block: 352687 duration=150431750 nS; ns/byte: 141782,
bandwidth=0.006726 MB/s
2016-12-08 21:30:22,183 DEBUG - Bytes in read #81: 16347 , block bytes: 712236,
remaining in block: 336340 duration=617356332 nS; ns/byte: 37765,
bandwidth=0.025252 MB/s
2016-12-08 21:30:22,183 DEBUG - Bytes in read #82: 1061 , block bytes: 713297,
remaining in block: 335279 duration=91692 nS; ns/byte: 86, bandwidth=11.035297
MB/s
2016-12-08 21:30:23,223 DEBUG - Bytes in read #83: 16347 , block bytes: 729644,
remaining in block: 318932 duration=1039467778 nS; ns/byte: 63587,
bandwidth=0.014998 MB/s
2016-12-08 21:30:23,223 DEBUG - Bytes in read #84: 1061 , block bytes: 730705,
remaining in block: 317871 duration=119170 nS; ns/byte: 112, bandwidth=8.490798
MB/s
2016-12-08 21:30:23,687 DEBUG - Bytes in read #85: 16347 , block bytes: 747052,
remaining in block: 301524 duration=463642449 nS; ns/byte: 28362,
bandwidth=0.033624 MB/s
2016-12-08 21:30:23,996 DEBUG - Bytes in read #86: 1061 , block bytes: 748113,
remaining in block: 300463 duration=308626480 nS; ns/byte: 290882,
bandwidth=0.003279 MB/s
2016-12-08 21:30:24,763 DEBUG - Bytes in read #87: 16347 , block bytes: 764460,
remaining in block: 284116 duration=766976410 nS; ns/byte: 46918,
bandwidth=0.020326 MB/s
2016-12-08 21:30:24,764 DEBUG - Bytes in read #88: 1061 , block bytes: 765521,
remaining in block: 283055 duration=94835 nS; ns/byte: 89, bandwidth=10.669568
MB/s
2016-12-08 21:30:26,695 DEBUG - Bytes in read #89: 16347 , block bytes: 781868,
remaining in block: 266708 duration=1930730977 nS; ns/byte: 118109,
bandwidth=0.008075 MB/s
2016-12-08 21:30:27,003 DEBUG - Bytes in read #90: 1061 , block bytes: 782929,
remaining in block: 265647 duration=307614364 nS; ns/byte: 289928,
bandwidth=0.003289 MB/s
2016-12-08 21:30:27,465 DEBUG - Bytes in read #91: 16347 , block bytes: 799276,
remaining in block: 249300 duration=461784239 nS; ns/byte: 28248,
bandwidth=0.033760 MB/s
2016-12-08 21:30:27,465 DEBUG - Bytes in read #92: 1061 , block bytes: 800337,
remaining in block: 248239 duration=62419 nS; ns/byte: 58, bandwidth=16.210584
MB/s
2016-12-08 21:30:28,333 DEBUG - Bytes in read #93: 16347 , block bytes: 816684,
remaining in block: 231892 duration=868005446 nS; ns/byte: 53098,
bandwidth=0.017960 MB/s
2016-12-08 21:30:28,895 DEBUG - Bytes in read #94: 1061 , block bytes: 817745,
remaining in block: 230831 duration=560647267 nS; ns/byte: 528414,
bandwidth=0.001805 MB/s
2016-12-08 21:30:29,762 DEBUG - Bytes in read #95: 16347 , block bytes: 834092,
remaining in block: 214484 duration=866587960 nS; ns/byte: 53012,
bandwidth=0.017990 MB/s
2016-12-08 21:30:29,864 DEBUG - Bytes in read #96: 1061 , block bytes: 835153,
remaining in block: 213423 duration=101564552 nS; ns/byte: 95725,
bandwidth=0.009963 MB/s
2016-12-08 21:30:30,379 DEBUG - Bytes in read #97: 16347 , block bytes: 851500,
remaining in block: 197076 duration=514754938 nS; ns/byte: 31489,
bandwidth=0.030286 MB/s
2016-12-08 21:30:30,688 DEBUG - Bytes in read #98: 1061 , block bytes: 852561,
remaining in block: 196015 duration=309420585 nS; ns/byte: 291631,
bandwidth=0.003270 MB/s
2016-12-08 21:30:31,457 DEBUG - Bytes in read #99: 16347 , block bytes: 868908,
remaining in block: 179668 duration=768580519 nS; ns/byte: 47016,
bandwidth=0.020284 MB/s
2016-12-08 21:30:31,458 DEBUG - Bytes in read #100: 1061 , block bytes: 869969,
remaining in block: 178607 duration=110392 nS; ns/byte: 104, bandwidth=9.165958
MB/s
2016-12-08 21:30:32,225 DEBUG - Bytes in read #101: 16347 , block bytes:
886316, remaining in block: 162260 duration=767381785 nS; ns/byte: 46943,
bandwidth=0.020315 MB/s
2016-12-08 21:30:32,226 DEBUG - Bytes in read #102: 1061 , block bytes: 887377,
remaining in block: 161199 duration=182729 nS; ns/byte: 172, bandwidth=5.537427
MB/s
2016-12-08 21:30:33,095 DEBUG - Bytes in read #103: 16347 , block bytes:
903724, remaining in block: 144852 duration=869101542 nS; ns/byte: 53165,
bandwidth=0.017938 MB/s
2016-12-08 21:30:33,247 DEBUG - Bytes in read #104: 1061 , block bytes: 904785,
remaining in block: 143791 duration=151712419 nS; ns/byte: 142990,
bandwidth=0.006670 MB/s
2016-12-08 21:30:33,711 DEBUG - Bytes in read #105: 16347 , block bytes:
921132, remaining in block: 127444 duration=463346639 nS; ns/byte: 28344,
bandwidth=0.033646 MB/s
2016-12-08 21:30:33,864 DEBUG - Bytes in read #106: 1061 , block bytes: 922193,
remaining in block: 126383 duration=152830422 nS; ns/byte: 144043,
bandwidth=0.006621 MB/s
2016-12-08 21:30:34,641 DEBUG - Bytes in read #107: 16347 , block bytes:
938540, remaining in block: 110036 duration=777184061 nS; ns/byte: 47542,
bandwidth=0.020059 MB/s
2016-12-08 21:30:34,642 DEBUG - Bytes in read #108: 1061 , block bytes: 939601,
remaining in block: 108975 duration=92668 nS; ns/byte: 87, bandwidth=10.919071
MB/s
2016-12-08 21:30:35,655 DEBUG - Bytes in read #109: 16347 , block bytes:
955948, remaining in block: 92628 duration=1013110625 nS; ns/byte: 61975,
bandwidth=0.015388 MB/s
2016-12-08 21:30:35,810 DEBUG - Bytes in read #110: 1061 , block bytes: 957009,
remaining in block: 91567 duration=154218951 nS; ns/byte: 145352,
bandwidth=0.006561 MB/s
2016-12-08 21:30:36,275 DEBUG - Bytes in read #111: 16347 , block bytes:
973356, remaining in block: 75220 duration=465093303 nS; ns/byte: 28451,
bandwidth=0.033520 MB/s
2016-12-08 21:30:36,276 DEBUG - Bytes in read #112: 1061 , block bytes: 974417,
remaining in block: 74159 duration=171536 nS; ns/byte: 161, bandwidth=5.898753
MB/s
2016-12-08 21:30:36,745 DEBUG - Bytes in read #113: 16347 , block bytes:
990764, remaining in block: 57812 duration=468606455 nS; ns/byte: 28666,
bandwidth=0.033268 MB/s
2016-12-08 21:30:36,745 DEBUG - Bytes in read #114: 1061 , block bytes: 991825,
remaining in block: 56751 duration=91092 nS; ns/byte: 85, bandwidth=11.107984
MB/s
2016-12-08 21:30:37,206 DEBUG - Bytes in read #115: 16347 , block bytes:
1008172, remaining in block: 40404 duration=460276607 nS; ns/byte: 28156,
bandwidth=0.033870 MB/s
2016-12-08 21:30:37,514 DEBUG - Bytes in read #116: 1061 , block bytes:
1009233, remaining in block: 39343 duration=307705385 nS; ns/byte: 290014,
bandwidth=0.003288 MB/s
2016-12-08 21:30:37,975 DEBUG - Bytes in read #117: 16347 , block bytes:
1025580, remaining in block: 22996 duration=461043201 nS; ns/byte: 28203,
bandwidth=0.033814 MB/s
2016-12-08 21:30:37,975 DEBUG - Bytes in read #118: 1061 , block bytes:
1026641, remaining in block: 21935 duration=74629 nS; ns/byte: 70,
bandwidth=13.558381 MB/s
2016-12-08 21:30:38,745 DEBUG - Bytes in read #119: 16347 , block bytes:
1042988, remaining in block: 5588 duration=768900663 nS; ns/byte: 47036,
bandwidth=0.020275 MB/s
2016-12-08 21:30:38,745 DEBUG - Bytes in read #120: 1061 , block bytes:
1044049, remaining in block: 4527 duration=161553 nS; ns/byte: 152,
bandwidth=6.263260 MB/s
2016-12-08 21:30:39,362 DEBUG - Bytes in read #121: 4527 , block bytes:
1048576, remaining in block: 0 duration=616755333 nS; ns/byte: 136239,
bandwidth=0.007000 MB/s
2016-12-08 21:30:39,363 INFO contract.ContractTestUtils
(ContractTestUtils.java:end(1441)) - Duration of Reading block 1 in 121 reads:
50,463,661,382 nS
2016-12-08 21:30:39,363 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) -
Bandwidth of block 1: 0.019816 MB/s:
2016-12-08 21:30:39,363 WARN (testTimeToOpenAndReadWholeFileBlocks(256)) -
Bandwidth 0.019816 too low on block 1: resetting connection
2016-12-08 21:30:39,366 DEBUG conn.DefaultClientConnection
(DefaultClientConnection.java:shutdown(166)) - Connection
0.0.0.0:57950<->54.231.169.2:443 shut down
2016-12-08 21:30:39,366 DEBUG conn.DefaultClientConnection
(DefaultClientConnection.java:close(182)) - Connection
0.0.0.0:57950<->54.231.169.2:443 closed
2016-12-08 21:30:39,366 DEBUG conn.PoolingClientConnectionManager
(PoolingClientConnectionManager.java:releaseConnection(274)) - Connection
released: [id: 1][route: {s}->https://landsat-pds.s3.amazonaws.com:443][total
kept alive: 0; route allocated: 0 of 25; total allocated: 0 of 25]
2016-12-08 21:30:39,366 DEBUG s3a.S3AFileSystem
(S3AInputStream.java:closeStream(463)) - Stream s3a://landsat-pds/scene_list.gz
aborted: reset(); streamPos=1048576, nextReadPos=1048576, request range
0-39278765 length=39278765
2016-12-08 21:30:39,367 DEBUG s3a.S3AFileSystem
(S3AInputStream.java:reopen(140)) - reopen(s3a://landsat-pds/scene_list.gz) for
read from new offset range[1048576-39278765], length=1048576,
streamPosition=1048576, nextReadPosition=1048576
2016-12-08 21:30:39,367 DEBUG amazonaws.request
(AmazonHttpClient.java:executeOneRequest(671)) - Sending Request: GET
https://landsat-pds.s3.amazonaws.com /scene_list.gz Headers: (User-Agent:
Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.10.6 Mac_OS_X/10.12.1
Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, Range:
bytes=1048576-39278765, Content-Type: application/x-www-form-urlencoded;
charset=utf-8, )
2016-12-08 21:30:39,367 DEBUG conn.PoolingClientConnectionManager
(PoolingClientConnectionManager.java:requestConnection(184)) - Connection
request: [route: {s}->https://landsat-pds.s3.amazonaws.com:443][total kept
alive: 0; route allocated: 0 of 25; total allocated: 0 of 25]
2016-12-08 21:30:39,368 DEBUG conn.PoolingClientConnectionManager
(PoolingClientConnectionManager.java:leaseConnection(218)) - Connection leased:
[id: 2][route: {s}->https://landsat-pds.s3.amazonaws.com:443][total kept alive:
0; route allocated: 1 of 25; total allocated: 1 of 25]
2016-12-08 21:30:39,394 DEBUG conn.DefaultClientConnectionOperator
(DefaultClientConnectionOperator.java:openConnection(176)) - Connecting to
landsat-pds.s3.amazonaws.com:443
2016-12-08 21:30:40,040 DEBUG protocol.RequestAddCookies
(RequestAddCookies.java:process(122)) - CookieSpec selected: default
2016-12-08 21:30:40,040 DEBUG protocol.RequestAuthCache
(RequestAuthCache.java:process(76)) - Auth cache not set in the context
2016-12-08 21:30:40,040 DEBUG protocol.RequestProxyAuthentication
(RequestProxyAuthentication.java:process(88)) - Proxy auth state: UNCHALLENGED
2016-12-08 21:30:40,040 DEBUG conn.DefaultClientConnection
(DefaultClientConnection.java:sendRequestHeader(276)) - Sending request: GET
/scene_list.gz HTTP/1.1
2016-12-08 21:30:40,041 DEBUG http.headers
(DefaultClientConnection.java:sendRequestHeader(280)) - >> GET /scene_list.gz
HTTP/1.1
2016-12-08 21:30:40,041 DEBUG http.headers
(DefaultClientConnection.java:sendRequestHeader(283)) - >> Host:
landsat-pds.s3.amazonaws.com
2016-12-08 21:30:40,041 DEBUG http.headers
(DefaultClientConnection.java:sendRequestHeader(283)) - >> Authorization: AWS
AKIAIYZ5JQOW3N5H6NPA:3c0hXBn3xs2erSymklPv8hV1MKM=
2016-12-08 21:30:40,041 DEBUG http.headers
(DefaultClientConnection.java:sendRequestHeader(283)) - >> User-Agent: Hadoop
2.8.0-SNAPSHOT, aws-sdk-java/1.10.6 Mac_OS_X/10.12.1
Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102
2016-12-08 21:30:40,041 DEBUG http.headers
(DefaultClientConnection.java:sendRequestHeader(283)) - >> Range:
bytes=1048576-39278765
2016-12-08 21:30:40,041 DEBUG http.headers
(DefaultClientConnection.java:sendRequestHeader(283)) - >> Date: Thu, 08 Dec
2016 21:30:39 GMT
2016-12-08 21:30:40,041 DEBUG http.headers
(DefaultClientConnection.java:sendRequestHeader(283)) - >> Content-Type:
application/x-www-form-urlencoded; charset=utf-8
2016-12-08 21:30:40,041 DEBUG http.headers
(DefaultClientConnection.java:sendRequestHeader(283)) - >> Connection:
Keep-Alive
2016-12-08 21:30:40,210 DEBUG conn.DefaultClientConnection
(DefaultClientConnection.java:receiveResponseHeader(261)) - Receiving response:
HTTP/1.1 206 Partial Content
2016-12-08 21:30:40,210 DEBUG http.headers
(DefaultClientConnection.java:receiveResponseHeader(264)) - << HTTP/1.1 206
Partial Content
2016-12-08 21:30:40,211 DEBUG http.headers
(DefaultClientConnection.java:receiveResponseHeader(267)) - << x-amz-id-2:
qKwcRkY1K8w22XWihP8NPtJWfcVbmX7jADNiNYk53DXSsbYOAxUobZwEpIPt0TN27ru9TBJNNA0=
2016-12-08 21:30:40,211 DEBUG http.headers
(DefaultClientConnection.java:receiveResponseHeader(267)) - <<
x-amz-request-id: CA85E25E70613150
2016-12-08 21:30:40,211 DEBUG http.headers
(DefaultClientConnection.java:receiveResponseHeader(267)) - << Date: Thu, 08
Dec 2016 21:30:41 GMT
2016-12-08 21:30:40,211 DEBUG http.headers
(DefaultClientConnection.java:receiveResponseHeader(267)) - << Last-Modified:
Thu, 08 Dec 2016 17:21:24 GMT
2016-12-08 21:30:40,211 DEBUG http.headers
(DefaultClientConnection.java:receiveResponseHeader(267)) - << ETag:
"6efe666c9bc2affbdba4f43dce9d0d55"
2016-12-08 21:30:40,211 DEBUG http.headers
(DefaultClientConnection.java:receiveResponseHeader(267)) - << Accept-Ranges:
bytes
2016-12-08 21:30:40,211 DEBUG http.headers
(DefaultClientConnection.java:receiveResponseHeader(267)) - << Content-Range:
bytes 1048576-39278764/39278765
2016-12-08 21:30:40,211 DEBUG http.headers
(DefaultClientConnection.java:receiveResponseHeader(267)) - << Content-Type:
application/octet-stream
2016-12-08 21:30:40,211 DEBUG http.headers
(DefaultClientConnection.java:receiveResponseHeader(267)) - << Content-Length:
38230189
2016-12-08 21:30:40,213 DEBUG http.headers
(DefaultClientConnection.java:receiveResponseHeader(267)) - << Server: AmazonS3
2016-12-08 21:30:40,213 DEBUG amazonaws.request
(AmazonHttpClient.java:logHeaderRequestId(836)) - x-amzn-RequestId: not
available
2016-12-08 21:30:40,214 DEBUG amazonaws.request
(AmazonHttpClient.java:handleResponse(1091)) - Received successful response:
206, AWS Request ID: CA85E25E70613150
2016-12-08 21:30:40,214 DEBUG amazonaws.request
(AmazonHttpClient.java:logResponseRequestId(856)) - AWS Request ID:
CA85E25E70613150
2016-12-08 21:30:40,364 DEBUG - Bytes in read #1: 16347 , block bytes: 16347,
remaining in block: 1032229 duration=997179473 nS; ns/byte: 61000,
bandwidth=0.015634 MB/s
2016-12-08 21:30:40,365 DEBUG - Bytes in read #2: 624 , block bytes: 16971,
remaining in block: 1031605 duration=101431 nS; ns/byte: 162,
bandwidth=5.866971 MB/s
2016-12-08 21:30:40,368 DEBUG - Bytes in read #3: 16347 , block bytes: 33318,
remaining in block: 1015258 duration=3318034 nS; ns/byte: 202,
bandwidth=4.698479 MB/s
2016-12-08 21:30:40,368 DEBUG - Bytes in read #4: 1061 , block bytes: 34379,
remaining in block: 1014197 duration=82636 nS; ns/byte: 77, bandwidth=12.244645
MB/s
2016-12-08 21:30:40,519 DEBUG - Bytes in read #5: 16347 , block bytes: 50726,
remaining in block: 997850 duration=150854582 nS; ns/byte: 9228,
bandwidth=0.103343 MB/s
2016-12-08 21:30:40,520 DEBUG - Bytes in read #6: 1061 , block bytes: 51787,
remaining in block: 996789 duration=76297 nS; ns/byte: 71, bandwidth=13.261969
MB/s
2016-12-08 21:30:40,522 DEBUG - Bytes in read #7: 16347 , block bytes: 68134,
remaining in block: 980442 duration=2337916 nS; ns/byte: 143,
bandwidth=6.668210 MB/s
2016-12-08 21:30:40,523 DEBUG - Bytes in read #8: 1061 , block bytes: 69195,
remaining in block: 979381 duration=687085 nS; ns/byte: 647, bandwidth=1.472669
MB/s
2016-12-08 21:30:40,672 DEBUG - Bytes in read #9: 16347 , block bytes: 85542,
remaining in block: 963034 duration=148946723 nS; ns/byte: 9111,
bandwidth=0.104666 MB/s
2016-12-08 21:30:40,673 DEBUG - Bytes in read #10: 1061 , block bytes: 86603,
remaining in block: 961973 duration=88030 nS; ns/byte: 82, bandwidth=11.494359
MB/s
2016-12-08 21:30:40,675 DEBUG - Bytes in read #11: 16347 , block bytes: 102950,
remaining in block: 945626 duration=1850448 nS; ns/byte: 113,
bandwidth=8.424832 MB/s
2016-12-08 21:30:40,675 DEBUG - Bytes in read #12: 1061 , block bytes: 104011,
remaining in block: 944565 duration=80706 nS; ns/byte: 76, bandwidth=12.537463
MB/s
2016-12-08 21:30:40,677 DEBUG - Bytes in read #13: 16347 , block bytes: 120358,
remaining in block: 928218 duration=1939042 nS; ns/byte: 118,
bandwidth=8.039905 MB/s
2016-12-08 21:30:40,677 DEBUG - Bytes in read #14: 1061 , block bytes: 121419,
remaining in block: 927157 duration=70915 nS; ns/byte: 66, bandwidth=14.268469
MB/s
2016-12-08 21:30:40,824 DEBUG - Bytes in read #15: 16347 , block bytes: 137766,
remaining in block: 910810 duration=146688052 nS; ns/byte: 8973,
bandwidth=0.106278 MB/s
2016-12-08 21:30:40,825 DEBUG - Bytes in read #16: 1061 , block bytes: 138827,
remaining in block: 909749 duration=135687 nS; ns/byte: 127, bandwidth=7.457225
MB/s
2016-12-08 21:30:40,827 DEBUG - Bytes in read #17: 16347 , block bytes: 155174,
remaining in block: 893402 duration=2159648 nS; ns/byte: 132,
bandwidth=7.218637 MB/s
2016-12-08 21:30:40,828 DEBUG - Bytes in read #18: 1061 , block bytes: 156235,
remaining in block: 892341 duration=81945 nS; ns/byte: 77, bandwidth=12.347897
MB/s
2016-12-08 21:30:40,830 DEBUG - Bytes in read #19: 16347 , block bytes: 172582,
remaining in block: 875994 duration=1960358 nS; ns/byte: 119,
bandwidth=7.952483 MB/s
2016-12-08 21:30:40,830 DEBUG - Bytes in read #20: 1061 , block bytes: 173643,
remaining in block: 874933 duration=80031 nS; ns/byte: 75, bandwidth=12.643206
MB/s
2016-12-08 21:30:40,833 DEBUG - Bytes in read #21: 16347 , block bytes: 189990,
remaining in block: 858586 duration=2730078 nS; ns/byte: 167,
bandwidth=5.710355 MB/s
2016-12-08 21:30:40,833 DEBUG - Bytes in read #22: 1061 , block bytes: 191051,
remaining in block: 857525 duration=82210 nS; ns/byte: 77, bandwidth=12.308095
MB/s
2016-12-08 21:30:40,836 DEBUG - Bytes in read #23: 16347 , block bytes: 207398,
remaining in block: 841178 duration=2466188 nS; ns/byte: 150,
bandwidth=6.321381 MB/s
2016-12-08 21:30:40,836 DEBUG - Bytes in read #24: 1061 , block bytes: 208459,
remaining in block: 840117 duration=75878 nS; ns/byte: 71, bandwidth=13.335202
MB/s
...
2016-12-08 21:30:41,456 DEBUG - Bytes in read #118: 1061 , block bytes:
1026635, remaining in block: 21941 duration=69393 nS; ns/byte: 65,
bandwidth=14.581420 MB/s
2016-12-08 21:30:41,459 DEBUG - Bytes in read #119: 16347 , block bytes:
1042982, remaining in block: 5594 duration=2944939 nS; ns/byte: 180,
bandwidth=5.293731 MB/s
2016-12-08 21:30:41,459 DEBUG - Bytes in read #120: 1061 , block bytes:
1044043, remaining in block: 4533 duration=68294 nS; ns/byte: 64,
bandwidth=14.816067 MB/s
2016-12-08 21:30:41,462 DEBUG - Bytes in read #121: 4533 , block bytes:
1048576, remaining in block: 0 duration=3124460 nS; ns/byte: 689,
bandwidth=1.383601 MB/s
2016-12-08 21:30:41,463 INFO contract.ContractTestUtils
(ContractTestUtils.java:end(1441)) - Duration of Reading block 2 in 121 reads:
2,095,902,133 nS
2016-12-08 21:30:41,911 INFO contract.ContractTestUtils
(ContractTestUtils.java:end(1441)) - Duration of Reading block 3 in 121 reads:
447,989,141 nS
2016-12-08 21:30:41,911 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) -
Bandwidth of block 3: 2.232197 MB/s:
2016-12-08 21:30:42,212 INFO contract.ContractTestUtils
(ContractTestUtils.java:end(1441)) - Duration of Reading block 4 in 121 reads:
300,567,979 nS
2016-12-08 21:30:42,212 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) -
Bandwidth of block 4: 3.327034 MB/s:
2016-12-08 21:30:42,445 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) -
Bandwidth of block 5: 4.304842 MB/s:
2016-12-08 21:30:42,641 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) -
Bandwidth of block 6: 5.102818 MB/s:
2016-12-08 21:30:43,709 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) -
Bandwidth of block 8: 1.389905 MB/s:
2016-12-08 21:30:44,705 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) -
Bandwidth of block 9: 1.003957 MB/s:
2016-12-08 21:30:45,020 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) -
Bandwidth of block 10: 3.184452 MB/s:
2016-12-08 21:30:45,269 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) -
Bandwidth of block 11: 4.012955 MB/s:
2016-12-08 21:30:45,608 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) -
Bandwidth of block 12: 2.951960 MB/s:
2016-12-08 21:30:45,886 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) -
Bandwidth of block 13: 3.598163 MB/s:
2016-12-08 21:30:46,212 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) -
Bandwidth of block 14: 3.070112 MB/s:
2016-12-08 21:30:46,542 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) -
Bandwidth of block 15: 3.029587 MB/s:
2016-12-08 21:30:46,858 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) -
Bandwidth of block 16: 3.175305 MB/s:
2016-12-08 21:30:47,173 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) -
Bandwidth of block 17: 3.175283 MB/s:
2016-12-08 21:30:47,494 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) -
Bandwidth of block 18: 3.111757 MB/s:
2016-12-08 21:30:47,809 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) -
Bandwidth of block 19: 3.182531 MB/s:
....
2016-12-08 21:30:53,746 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) -
Bandwidth of block 37: 2.895820 MB/s:
2016-12-08 21:30:53,746 INFO contract.ContractTestUtils
(ContractTestUtils.java:end(1441)) - Duration of Time to read 39278765 bytes in
37 blocks: 64,847,176,023 nS
2016-12-08 21:30:53,746 INFO (logStreamStatistics(366)) - Stream Statistics
StreamStatistics{OpenOperations=2, CloseOperations=1, Closed=0, Aborted=1,
SeekOperations=0, ReadExceptions=0, ForwardSeekOperations=0,
BackwardSeekOperations=0, BytesSkippedOnSeek=0, BytesBackwardsOnSeek=0,
BytesRead=38797312, BytesRead excluding skipped=38797312, ReadOperations=2833,
ReadFullyOperations=0, ReadsIncomplete=2796, BytesReadInClose=0,
BytesDiscardedInAbort=38230189}
2016-12-08 21:30:53,746 WARN (testTimeToOpenAndReadWholeFileBlocks(269)) -
Reset the connection 1 time(s)
2016-12-08 21:30:53,747 INFO contract.AbstractFSContractTestBase
(AbstractFSContractTestBase.java:describe(255)) - cleanup
2016-12-08 21:30:53,747 DEBUG conn.DefaultClientConnection
(DefaultClientConnection.java:shutdown(166)) - Connection
0.0.0.0:57959<->54.231.169.22:443 shut down
2016-12-08 21:30:53,747 DEBUG conn.DefaultClientConnection
(DefaultClientConnection.java:close(182)) - Connection
0.0.0.0:57959<->54.231.169.22:443 closed
2016-12-08 21:30:53,747 DEBUG conn.PoolingClientConnectionManager
(PoolingClientConnectionManager.java:releaseConnection(274)) - Connection
released: [id: 2][route: {s}->https://landsat-pds.s3.amazonaws.com:443][total
kept alive: 0; route allocated: 0 of 25; total allocated: 0 of 25]
2016-12-08 21:30:53,747 DEBUG s3a.S3AFileSystem
(S3AInputStream.java:closeStream(463)) - Stream s3a://landsat-pds/scene_list.gz
aborted: close() operation; streamPos=38797312, nextReadPos=38797312, request
range 1048576-39278765 length=39278765
{code}
> ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks
> performance on branch-2.8 awful
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-13871
> URL: https://issues.apache.org/jira/browse/HADOOP-13871
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha2
> Environment: landsat bucket from the UK
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-13871-001.patch,
> org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance-output.txt
>
>
> The ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks takes
> 15s on branch-2, but is now taking minutes.
> This is a regression, and it's surfacing on some internal branches too. Even
> where the code hasn't changed. -It does not happen on branch-2, which has a
> later SDK.-
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
