Hi ,
Thanks for pointing the docs, actually i was missing to create local file to
copy to HDFS after create. My issues exactly was with having knox server behind
the LB . The location header which it returns in first pass was giving my knox
port instead of LB URl port 443. So some documentation says use front-end url
(>gateway.frontend.url) parameter . Some how its not respecting correctly when
having Nginx LB config.
Any reference/documentation for having Nginx as tcp LB config for knox. So that
SSL offload will happen at knox server and above webhdfs create works well.
\Rajesh
On Friday, 5 February 2016 8:18 PM, Kevin Minder
<[email protected]> wrote:
Rajesh,From your message I'm not clear on what didn't work? It seems you
might not actually be familiar with the WebHDFS file creation process.See this
in the Knox User's Guide:
http://knox.apache.org/books/knox-0-6-0/user-guide.html#WebHDFS+via+cURLIn
particular this step:
http://knox.apache.org/books/knox-0-6-0/user-guide.html#WebHDFS+via+cURLKevin.
From: Rajesh Chandramohan <[email protected]>
Reply-To: "[email protected]" <[email protected]>, Rajesh Chandramohan
<[email protected]>
Date: Friday, February 5, 2016 at 7:19 AM
To: "[email protected]" <[email protected]>, Larry Mccay
<[email protected]>, Christopher Jackson
<[email protected]>, Kevin Minder <[email protected]>
Subject: Knox Create command doesn't work
Hi knox-Team,
We were configured Knox-.0.6 to access secured hadoop . We could do LISTSTATUS
but same time CREATE does't works. Do we need to have some extra config or
arguments to pass to make CREATE work
LISTSTATUS : Worked curl -ik --verbose --negotiate -u
:https://knox01-88401.vmmachine.com:8443/gateway/cluster/webhdfs/v1/tmp/?op=LISTSTATUS
CREATE : curl -ik --verbose --negotiate -u : -X PUT
https://knox01-88401.vmmachine.com:8443/gateway/cluster/webhdfs/v1/tmp/testrajesh2?op=CREATE
< Location:
https://knox01-88401.vmmachine.com:8443/gateway/cluster/webhdfs/data/v1/webhdfs/v1/tmp/testrajesh2?_=AAAACAAAABAAAAEQks7kK_7JR3jJILg1FcBujP6NeCP6Sb6nl_uR2iSSEKHy6QhNfv9CZItrs5Uvz0MaaCKkyYgVZ-1CKXO7IddvG2zA3hYHT4MjMWjzamQqMVZBhJWRa1wkw_tqVQPALRkg7MDlO9pVBNbkvW038iGXFeMXCI3B0Plr5iYpIHWWafIe-H2lR29JEyRhIiTMySRn1lLQs-L6R7zoPVxmt6UVSy3bwy9jpXByoA6jv08F72jMhNeXy6K-w8A-j5rk5Sk2RJ63IDuy1-QCMQpRy9yYJa7It4fJeTJBVXqOy4guuIDG0QKGp5_gKVE3Z7e0ZIiz5QDXuvJ3jbF5dl-dDFDwYDWDSkt9MLgwv8iXC-CtcnuO0abgpZjR06qUtjUtzqBszjgj2JagwLALocation:
https://knox01-88401.vmmachine.com:8443/gateway/cluster/webhdfs/data/v1/webhdfs/v1/tmp/testrajesh2?_=AAAACAAAABAAAAEQks7kK_7JR3jJILg1FcBujP6NeCP6Sb6nl_uR2iSSEKHy6QhNfv9CZItrs5Uvz0MaaCKkyYgVZ-1CKXO7IddvG2zA3hYHT4MjMWjzamQqMVZBhJWRa1wkw_tqVQPALRkg7MDlO9pVBNbkvW038iGXFeMXCI3B0Plr5iYpIHWWafIe-H2lR29JEyRhIiTMySRn1lLQs-L6R7zoPVxmt6UVSy3bwy9jpXByoA6jv08F72jMhNeXy6K-w8A-j5rk5Sk2RJ63IDuy1-QCMQpRy9yYJa7It4fJeTJBVXqOy4guuIDG0QKGp5_gKVE3Z7e0ZIiz5QDXuvJ3jbF5dl-dDFDwYDWDSkt9MLgwv8iXC-CtcnuO0abgpZjR06qUtjUtzqBszjgj2JagwLA<
Server: Jetty(6.1.26)Server: Jetty(6.1.26)< Content-Type:
application/octet-streamContent-Type: application/octet-stream< Content-Length:
0Content-Length: 0--------
DEBUG LOGS:016-02-05 04:57:24,592 DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onResponseReceived(116)) -
http-outgoing-1 << Content-Type: application/octet-stream2016-02-05
04:57:24,592 DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onResponseReceived(116)) -
http-outgoing-1 <<
Location:http://phx4b03c-8577.DATANODE.hst.com:1006/webhdfs/v1/tmp/testrajesh2?op=CREATE&delegation=KgAGYXBwbW9uBmJfa25veAZiX2tub3iKAVKxSz7DigFS1VfCw465P44FDRSpCmdF27xvPk0oNpWnhNrBlCQhPxJXRUJIREZTIGRlbGVnYXRpb24TMTAuMTE1LjIwMS4xMDM6ODAyMA&namenoderpcaddress=hera-nn-ha&overwrite=false2016-02-05
04:57:24,592 DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onResponseReceived(116)) -
http-outgoing-1 << Content-Length: 02016-02-05 04:57:24,592 DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onResponseReceived(116)) -
http-outgoing-1 << Server: Jetty(6.1.26)2016-02-05 04:57:24,592 DEBUG
execchain.MainClientExec (MainClientExec.java:execute(267)) - Connection can be
kept alive indefinitely2016-02-05 04:57:24,593 DEBUG hadoop.gateway
(DefaultDispatch.java:executeOutboundRequest(136)) - Dispatch response status:
3072016-02-05 04:57:24,740 DEBUG conn.DefaultClientConnection
(DefaultClientConnection.java:close(176)) - Connection
0.0.0.0:51485<->10.115.201.104:50080 closed2016-02-05 04:57:24,770 DEBUG
hadoop.gateway (UrlRewriteProcessor.java:rewrite(157)) - Rewrote URL:
http://phx4b03c-8577.DATANODE.hst.com:1006/webhdfs/v1/tmp/testrajesh2?op=CREATE&delegation=KgAGYXBwbW9uBmJfa25veAZiX2tub3iKAVKxSz7DigFS1VfCw465P44FDRSpCmdF27xvPk0oNpWnhNrBlCQhPxJXRUJIREZTIGRlbGVnYXRpb24TMTAuMTE1LjIwMS4xMDM6ODAyMA&namenoderpcaddress=hera-phx-nn-ha&overwrite=false,
direction: OUT via explicit rule:
WEBHDFS/webhdfs/outbound/namenode/headers/location to
URL:https://knox01-88401.vmmachine.com:8443/gateway/cluster/webhdfs/data/v1/webhdfs/v1/tmp/testrajesh2?_=AAAACAAAABAAAAEQgMKetk9z9vM-09d0e2tyobify1o_bvypEhXCyBU9c_xYlgjSb9aOsZW1IgPpfoSdSpYRCuHvjjAWPaei5TIwkX0akEwWxVQWcGMq94jHnU7T8B4mAoENC6xGoND0E7_pFBaLDO9SrilrlZUZLDKiKcgEGqo9SRY6oRO47WrSqXW0bFpWc-TYkRaemfNsu2fSojKxwZnheI_8M0AWxBI0N2EFfGlshO0k-zW1Og8q0XthYyavexCJxvuZ9tbPSs-roC_mxl-LlUyKAeS0KF45TlFoliAjtoQcxs3sbRoRe7abpRdJWCadMgDBRdR2fDMzg16A2hxs1XQ1uPSx2WQeqwNqqhKzL-6o7orNPLej1wVokXx3BhwubK0SIAoMhbvMadPQ-S8H5VE2016-02-05
04:57:24,792 DEBUG conn.PoolingHttpClientConnectionManager
(PoolingHttpClientConnectionManager.java:releaseConnection(281)) - Connection
[id: 1][route: {}->http:///cluster-namenode.vip.com:50080] can be kept alive
indefinitely2016-02-05 04:57:24,793 DEBUG
conn.PoolingHttpClientConnectionManager
(PoolingHttpClientConnectionManager.java:releaseConnection(287)) - Connection
released: [id: 1][route: {}->http://cluster-namenode.vip.com:50080][total kept
alive: 2; route allocated: 1 of 2; total allocated: 2 of 20]
\Rajesh
