Using Knox 0.6.0 with hadoop 2.7.1.
I am using the Hadoop FileSystem api but I have made some very minor
modifications to the WebHDFSFileSystem class:
1) I hardcoded the username/password credentials for the server such that
the connection to the server succeeds.
2) I also changed the default path for the connections such that the URL
path is
https://<HOST>:<PORT>/*gateway/default/*webhdfs/v1/<PATH>
instead of
https://<HOST>:<PORT>/webhdfs/v1/<PATH>
With these changes, I am able to successfully write to the server.
However, I cannot read. I get the following exception:
java.io.IOException: Content-Length is missing: {null=[HTTP/1.1 200 OK],
Server=[Jetty(8.1.14.v20131031)], Access-Control-Allow-Origin=[*],
Access-Control-Allow-Methods=[GET], Connection=[close],
Set-Cookie=[JSESSIONID=eknosvics9ou1s6ttv2kbfscl;Path=/gateway/default;Secure;HttpOnly],
Expires=[Thu, 01 Jan 1970 00:00:00 GMT],
Content-Type=[application/octet-stream]}
at
org.apache.hadoop.hdfs.web.ByteRangeInputStream.getStreamLength(ByteRangeInputStream.java:153)
at
org.apache.hadoop.hdfs.web.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:131)
at
org.apache.hadoop.hdfs.web.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:105)
at
org.apache.hadoop.hdfs.web.ByteRangeInputStream.<init>(ByteRangeInputStream.java:90)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$OffsetUrlInputStream.<init>(WebHdfsFileSystem.java:1275)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.open(WebHdfsFileSystem.java:1189)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
Comparing the headers between Knox 0.6 and Knox 0.5 shows that in v0.5 the
Content header *is* sent normally, but missing in the response to the
client from Knox 0.6.
I have turned on debugging in the Knox server and what is interesting is
that the trace shows that the file is read successfully and the right
headers seem to be generated, as is shown in the following excerpt from
gateway.log:
2015-11-27 23:22:48,757 DEBUG conn.HttpClientConnectionOperator
(HttpClientConnectionOperator.java:connect(129)) - Connection established
10.143.177.241:57593<->10.143.177.15:50075
2015-11-27 23:22:48,757 DEBUG execchain.MainClientExec
(MainClientExec.java:execute(238)) - Executing request GET
/webhdfs/v1/user/admin/hdfs_test_2015_20151127_182243.txt?user.name=admin&namenoderpcaddress=ehaascluster&buffersize=4096&offset=0&op=OPEN
HTTP/1.1
2015-11-27 23:22:48,758 DEBUG execchain.MainClientExec
(MainClientExec.java:execute(243)) - Target auth state: UNCHALLENGED
2015-11-27 23:22:48,758 DEBUG execchain.MainClientExec
(MainClientExec.java:execute(249)) - Proxy auth state: UNCHALLENGED
2015-11-27 23:22:48,758 DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onRequestSubmitted(124)) -
http-outgoing-16 >> GET
/webhdfs/v1/user/admin/hdfs_test_2015_20151127_182243.txt?user.name=admin&namenoderpcaddress=ehaascluster&buffersize=4096&offset=0&op=OPEN
HTTP/1.1
2015-11-27 23:22:48,758 DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onRequestSubmitted(127)) -
http-outgoing-16 >> Accept: text/html, image/gif, image/jpeg, *; q=.2, */*;
q=.2
2015-11-27 23:22:48,758 DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onRequestSubmitted(127)) -
http-outgoing-16 >> Connection: keep-alive
2015-11-27 23:22:48,759 DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onRequestSubmitted(127)) -
http-outgoing-16 >> User-Agent: Java/1.8.0
2015-11-27 23:22:48,759 DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onRequestSubmitted(127)) -
http-outgoing-16 >> Host: sampleserver.com:50075
2015-11-27 23:22:48,759 DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onRequestSubmitted(127)) -
http-outgoing-16 >> Accept-Encoding: gzip,deflate
2015-11-27 23:22:48,759 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 >> "GET
/webhdfs/v1/user/admin/hdfs_test_2015_20151127_182243.txt?user.name=admin&namenoderpcaddress=ehaascluster&buffersize=4096&offset=0&op=OPEN
HTTP/1.1[\r][\n]"
2015-11-27 23:22:48,759 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 >> "Accept: text/html, image/gif, image/jpeg, *; q=.2,
*/*; q=.2[\r][\n]"
2015-11-27 23:22:48,759 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 >> "Connection: keep-alive[\r][\n]"
2015-11-27 23:22:48,760 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 >> "User-Agent: Java/1.8.0[\r][\n]"
2015-11-27 23:22:48,760 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 >> "Host: sampleserver.com:50075[\r][\n]"
2015-11-27 23:22:48,760 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 >> "Accept-Encoding: gzip,deflate[\r][\n]"
2015-11-27 23:22:48,760 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 >> "[\r][\n]"
2015-11-27 23:22:48,765 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 << "HTTP/1.1 200 OK[\r][\n]"
2015-11-27 23:22:48,766 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 << "Access-Control-Allow-Methods: GET[\r][\n]"
2015-11-27 23:22:48,766 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 << "Access-Control-Allow-Origin: *[\r][\n]"
2015-11-27 23:22:48,766 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 << "Content-Type: application/octet-stream[\r][\n]"
2015-11-27 23:22:48,766 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 << "Connection: close[\r][\n]"
2015-11-27 23:22:48,767 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 << "Content-Length: 450[\r][\n]"
2015-11-27 23:22:48,767 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 << "[\r][\n]"
2015-11-27 23:22:48,767 DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onResponseReceived(113)) -
http-outgoing-16 << HTTP/1.1 200 OK
2015-11-27 23:22:48,767 DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onResponseReceived(116)) -
http-outgoing-16 << Access-Control-Allow-Methods: GET
2015-11-27 23:22:48,767 DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onResponseReceived(116)) -
http-outgoing-16 << Access-Control-Allow-Origin: *
2015-11-27 23:22:48,768 DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onResponseReceived(116)) -
http-outgoing-16 << Content-Type: application/octet-stream
2015-11-27 23:22:48,768 DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onResponseReceived(116)) -
http-outgoing-16 << Connection: close
2015-11-27 23:22:48,768 DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onResponseReceived(116)) -
http-outgoing-16 << Content-Length: 450
2015-11-27 23:22:48,769 DEBUG hadoop.gateway
(DefaultDispatch.java:executeOutboundRequest(136)) - Dispatch response
status: 200
2015-11-27 23:22:48,769 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 << "HDFS and knox read and write test New LINE 0[\n]"
2015-11-27 23:22:48,769 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 << "HDFS and knox read and write test New LINE 1[\n]"
2015-11-27 23:22:48,769 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 << "HDFS and knox read and write test New LINE 2[\n]"
2015-11-27 23:22:48,769 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 << "HDFS and knox read and write test New LINE 3[\n]"
2015-11-27 23:22:48,770 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 << "HDFS and knox read and write test New LINE 4[\n]"
2015-11-27 23:22:48,770 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 << "HDFS and knox read and write test New LINE 5[\n]"
2015-11-27 23:22:48,770 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 << "HDFS and knox read and write test New LINE 6[\n]"
2015-11-27 23:22:48,770 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 << "HDFS and knox read and write test New LINE 7[\n]"
2015-11-27 23:22:48,770 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 << "HDFS and knox read and write test New LINE 8[\n]"
2015-11-27 23:22:48,771 DEBUG http.wire (Wire.java:wire(72)) -
http-outgoing-16 << "HDFS and knox read and write test New LINE 9[\n]"
2015-11-27 23:22:48,771 DEBUG conn.DefaultManagedHttpClientConnection
(LoggingManagedHttpClientConnection.java:shutdown(87)) - http-outgoing-16:
Shutdown connection
2015-11-27 23:22:48,771 DEBUG execchain.MainClientExec
(ConnectionHolder.java:abortConnection(126)) - Connection discarded
2015-11-27 23:22:48,771 DEBUG conn.DefaultManagedHttpClientConnection
(LoggingManagedHttpClientConnection.java:close(79)) - http-outgoing-16:
Close connection
2015-11-27 23:22:48,771 DEBUG conn.PoolingHttpClientConnectionManager
(PoolingHttpClientConnectionManager.java:releaseConnection(287)) -
Connection released: [id: 16][route: {}->http://sampleserver.com:50075][total
kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
2015-11-27 23:22:48,772 DEBUG nio.ssl (SslConnection.java:process(347)) -
[Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection@172f1603 SSL
NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false
{AsyncHttpConnection@c8a8734,g=HttpGenerator{s=2,h=307,b=450,c=-1},p=HttpParser{s=-5,l=10,c=0},r=1}
NOT_HANDSHAKING filled=0/0 flushed=0/0
2015-11-27 23:22:48,772 DEBUG nio.ssl (SslConnection.java:wrap(462)) -
[Session-1, SSL_NULL_WITH_NULL_NULL] wrap OK NOT_HANDSHAKING consumed=307
produced=341
2015-11-27 23:22:48,772 DEBUG nio.ssl (SslConnection.java:process(347)) -
[Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection@172f1603 SSL
NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false
{AsyncHttpConnection@c8a8734,g=HttpGenerator{s=2,h=0,b=450,c=-1},p=HttpParser{s=-5,l=10,c=0},r=1}
NOT_HANDSHAKING filled=0/0 flushed=341/0
2015-11-27 23:22:48,773 DEBUG nio.ssl (SslConnection.java:process(347)) -
[Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection@172f1603 SSL
NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false
{AsyncHttpConnection@c8a8734,g=HttpGenerator{s=2,h=0,b=450,c=-1},p=HttpParser{s=-5,l=10,c=0},r=1}
NOT_HANDSHAKING filled=0/0 flushed=0/0
2015-11-27 23:22:48,773 DEBUG nio.ssl (SslConnection.java:process(347)) -
[Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection@172f1603 SSL
NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false
{AsyncHttpConnection@c8a8734,g=HttpGenerator{s=2,h=0,b=450,c=-1},p=HttpParser{s=-5,l=10,c=0},r=1}
NOT_HANDSHAKING filled=0/0 flushed=0/0
2015-11-27 23:22:48,773 DEBUG nio.ssl (SslConnection.java:wrap(462)) -
[Session-1, SSL_NULL_WITH_NULL_NULL] wrap OK NOT_HANDSHAKING consumed=450
produced=522
2015-11-27 23:22:48,773 DEBUG nio.ssl (SslConnection.java:process(347)) -
[Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection@172f1603 SSL
NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false
{AsyncHttpConnection@c8a8734,g=HttpGenerator{s=2,h=0,b=0,c=-1},p=HttpParser{s=-5,l=10,c=0},r=1}
NOT_HANDSHAKING filled=0/0 flushed=522/0
2015-11-27 23:22:48,774 DEBUG nio.ssl (SslConnection.java:process(347)) -
[Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection@172f1603 SSL
NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false
{AsyncHttpConnection@c8a8734,g=HttpGenerator{s=2,h=0,b=0,c=-1},p=HttpParser{s=-5,l=10,c=0},r=1}
NOT_HANDSHAKING filled=0/0 flushed=0/0
2015-11-27 23:22:48,774 DEBUG server.Server (Server.java:handle(367)) -
RESPONSE
/gateway/default/webhdfs/data/v1/webhdfs/v1/user/admin/hdfs_test_2015_20151127_182243.txt
200 handled=true
Is there something I'm missing? How come the final response from Knox does
not contain the content-length header when it seems the response from
Hadoop does?
--
-Natasha D'Silva
