jaehoon ko created KNOX-475:
-------------------------------
Summary: "Hit replay buffer max limit" when uploading greater than
4KB file to HDFS
Key: KNOX-475
URL: https://issues.apache.org/jira/browse/KNOX-475
Project: Apache Knox
Issue Type: Bug
Components: Server
Affects Versions: 0.4.0
Environment: CentOS 6.5
HttpFS 2.4.0, Kerberized
Oracle JDK 1.7.0_45 64bit
Reporter: jaehoon ko
My cluster has a KNOX gateway that forwards WebHDFS REST API to HttpFS. HttpFS
and HDFS are Kerberized.
If I try uploading a file bigger than 4K, I get 500 error.
{code}
$ curl -u cersei:cersei -ik -X PUT
"https://node-02-04:8443/gateway/gc/webhdfs/v1/user/cersei/bigfile?op=create";
HTTP/1.1 307 Temporary Redirect
Set-Cookie:
JSESSIONID=wlsp2k09p1mj15i0n1tfzdai0;Path=/gateway/gc;Secure;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Server: Apache-Coyote/1.1
Location:
https://node-02-04:8443/gateway/gc/webhdfs/data/v1/webhdfs/v1/user/cersei/bigfile?_=AAAACAAAABAAAABQ0N5SioA7O6sdYwm-BuvUtz3Ov6BvONMosqdDZf865RRUEsOmLDbvjh6vwMCr9kRebZ-_98ksmWnbOQkEMDbbZzYD_vrSCLTbGS9fkz1OLSqd4dOUaoIjZDsXj9QDgdbwOa6BPsjSUmg
Date: Tue, 02 Dec 2014 11:01:26 GMT
Content-Type: application/json
Content-Length: 0
curl -ik -H "Content-Type:application/octet-stream" -u cersei:cersei -T bigfile
-X PUT
"https://node-02-04:8443/gateway/gc/webhdfs/data/v1/webhdfs/v1/user/cersei/bigfile?_=AAAACAAAABAAAABQ0N5SioA7O6sdYwm-BuvUtz3Ov6BvONMosqdDZf865RRUEsOmLDbvjh6vwMCr9kRebZ-_98ksmWnbOQkEMDbbZzYD_vrSCLTbGS9fkz1OLSqd4dOUaoIjZDsXj9QDgdbwOa6BPsjSUmg";
HTTP/1.1 100 Continue
HTTP/1.1 500 Server Error
Set-Cookie:
JSESSIONID=1t8dm6cn9rlw31mzureff6nqum;Path=/gateway/gc;Secure;HttpOnly
Content-Length: 0
Server: Jetty(8.1.14.v20131031)
{code}
Knox log is:
{panel}
2014-12-02 20:01:26,411 INFO realm.AuthorizingRealm
(AuthorizingRealm.java:getAuthorizationCacheLazy(248)) - No cache or
cacheManager properties have been set. Authorization cache cannot be obtained.
2014-12-02 20:02:08,094 INFO realm.AuthorizingRealm
(AuthorizingRealm.java:getAuthorizationCacheLazy(248)) - No cache or
cacheManager properties have been set. Authorization cache cannot be obtained.
2014-12-02 20:02:08,395 INFO client.DefaultHttpClient
(DefaultRequestDirector.java:tryExecute(726)) - I/O exception
(java.io.IOException) caught when processing request: Hit replay buffer max
limit
2014-12-02 20:02:08,395 INFO client.DefaultHttpClient
(DefaultRequestDirector.java:tryExecute(733)) - Retrying request
2014-12-02 20:02:08,405 INFO client.DefaultHttpClient
(DefaultRequestDirector.java:tryExecute(726)) - I/O exception
(java.io.IOException) caught when processing request: Hit replay buffer max
limit
2014-12-02 20:02:08,407 INFO client.DefaultHttpClient
(DefaultRequestDirector.java:tryExecute(733)) - Retrying request
2014-12-02 20:02:08,415 INFO client.DefaultHttpClient
(DefaultRequestDirector.java:tryExecute(726)) - I/O exception
(java.io.IOException) caught when processing request: Hit replay buffer max
limit
2014-12-02 20:02:08,416 INFO client.DefaultHttpClient
(DefaultRequestDirector.java:tryExecute(733)) - Retrying request
2014-12-02 20:02:08,426 WARN hadoop.gateway
(HttpClientDispatch.java:executeRequest(124)) - Connection exception
dispatching request:
http://node-02-04.gc.net:14000/webhdfs/v1/user/cersei/bigfile?op=CREATE&data=true&doAs=cersei
java.io.IOException: Hit replay buffer max limit
java.io.IOException: Hit replay buffer max limit
at
org.apache.hadoop.gateway.dispatch.CappedBufferHttpEntity$ReplayStream.read(CappedBufferHttpEntity.java:143)
at java.io.InputStream.read(InputStream.java:101)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
at
org.apache.hadoop.gateway.dispatch.CappedBufferHttpEntity.writeTo(CappedBufferHttpEntity.java:93)
at
org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98)
at
org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
at
org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:122)
at
org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:271)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:197)
at
org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:257)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at
org.apache.hadoop.gateway.dispatch.HttpClientDispatch.executeKerberosDispatch(HttpClientDispatch.java:193)
at
org.apache.hadoop.gateway.dispatch.HttpClientDispatch.executeRequest(HttpClientDispatch.java:120)
at
org.apache.hadoop.gateway.dispatch.HttpClientDispatch.doPut(HttpClientDispatch.java:273)
at
org.apache.hadoop.gateway.dispatch.AbstractGatewayDispatch$PutAdapter.doMethod(AbstractGatewayDispatch.java:148)
at
org.apache.hadoop.gateway.dispatch.AbstractGatewayDispatch.doFilter(AbstractGatewayDispatch.java:64)
at
org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
at
org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter.doFilter(UrlRewriteServletFilter.java:60)
at
org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
at
org.apache.hadoop.gateway.filter.security.AbstractIdentityAssertionFilter.doFilterInternal(AbstractIdentityAssertionFilter.java:209)
at
org.apache.hadoop.gateway.filter.security.AbstractIdentityAssertionFilter.continueChainAsPrincipal(AbstractIdentityAssertionFilter.java:157)
at
org.apache.hadoop.gateway.identityasserter.filter.IdentityAsserterFilter.doFilter(IdentityAsserterFilter.java:55)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
at
org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:93)
at
org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:90)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:126)
at
org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:77)
at
org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at
org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at
org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at
org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter.doFilter(ShiroSubjectIdentityAdapter.java:74)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
at
org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
at
org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at
org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at
org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at
org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at
org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at
org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at
org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at
org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at
org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
at
org.apache.hadoop.gateway.filter.ResponseCookieFilter.doFilter(ResponseCookieFilter.java:38)
at
org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
at
org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:128)
at
org.apache.hadoop.gateway.GatewayServlet.service(GatewayServlet.java:117)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:651)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:701)
2014-12-02 20:02:08,433 ERROR hadoop.gateway
(AbstractGatewayFilter.java:doFilter(62)) - Failed to execute filter:
java.io.IOException: Service connectivity error.
2014-12-02 20:02:08,433 ERROR hadoop.gateway
(AbstractGatewayFilter.java:doFilter(62)) - Failed to execute filter:
java.io.IOException: Service connectivity error.
2014-12-02 20:02:08,434 ERROR hadoop.gateway
(AbstractGatewayFilter.java:doFilter(65)) - Failed to execute filter:
javax.servlet.ServletException: org.apache.shiro.subject.ExecutionException:
java.security.PrivilegedActionException: java.io.IOException: Service
connectivity error.
2014-12-02 20:02:08,434 ERROR hadoop.gateway (GatewayFilter.java:doFilter(134))
- Failed to execute filter: javax.servlet.ServletException:
org.apache.shiro.subject.ExecutionException:
java.security.PrivilegedActionException: java.io.IOException: Service
connectivity error.
2014-12-02 20:02:08,435 ERROR hadoop.gateway (GatewayServlet.java:service(122))
- Failed to execute filter: javax.servlet.ServletException:
org.apache.shiro.subject.ExecutionException:
java.security.PrivilegedActionException: java.io.IOException: Service
connectivity error.
2014-12-02 20:02:08,435 WARN servlet.ServletHandler
(ServletHandler.java:doHandle(536)) -
javax.servlet.ServletException: org.apache.shiro.subject.ExecutionException:
java.security.PrivilegedActionException: java.io.IOException: Service
connectivity error.
at
org.apache.shiro.web.servlet.AdviceFilter.cleanup(AdviceFilter.java:196)
at
org.apache.shiro.web.filter.authc.AuthenticatingFilter.cleanup(AuthenticatingFilter.java:155)
at
org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:148)
at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at
org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at
org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at
org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at
org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at
org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at
org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at
org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
at
org.apache.hadoop.gateway.filter.ResponseCookieFilter.doFilter(ResponseCookieFilter.java:38)
at
org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
at
org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:128)
at
org.apache.hadoop.gateway.GatewayServlet.service(GatewayServlet.java:117)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:651)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:701)
Caused by: org.apache.shiro.subject.ExecutionException:
java.security.PrivilegedActionException: java.io.IOException: Service
connectivity error.
at
org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:385)
at
org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter.doFilter(ShiroSubjectIdentityAdapter.java:74)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
at
org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
at
org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at
org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
... 42 more
Caused by: java.security.PrivilegedActionException: java.io.IOException:
Service connectivity error.
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:126)
at
org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:77)
at
org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at
org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at
org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
... 48 more
Caused by: java.io.IOException: Service connectivity error.
at
org.apache.hadoop.gateway.dispatch.HttpClientDispatch.executeRequest(HttpClientDispatch.java:126)
at
org.apache.hadoop.gateway.dispatch.HttpClientDispatch.doPut(HttpClientDispatch.java:273)
at
org.apache.hadoop.gateway.dispatch.AbstractGatewayDispatch$PutAdapter.doMethod(AbstractGatewayDispatch.java:148)
at
org.apache.hadoop.gateway.dispatch.AbstractGatewayDispatch.doFilter(AbstractGatewayDispatch.java:64)
at
org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
at
org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter.doFilter(UrlRewriteServletFilter.java:60)
at
org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
at
org.apache.hadoop.gateway.filter.security.AbstractIdentityAssertionFilter.doFilterInternal(AbstractIdentityAssertionFilter.java:209)
at
org.apache.hadoop.gateway.filter.security.AbstractIdentityAssertionFilter.continueChainAsPrincipal(AbstractIdentityAssertionFilter.java:157)
at
org.apache.hadoop.gateway.identityasserter.filter.IdentityAsserterFilter.doFilter(IdentityAsserterFilter.java:55)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
at
org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:93)
at
org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:90)
... 55 more
2014-12-02 20:02:08,451 WARN servlet.ServletHandler
(ServletHandler.java:doHandle(561)) -
/gateway/gc/webhdfs/data/v1/webhdfs/v1/user/cersei/bigfile
org.apache.shiro.subject.ExecutionException:
java.security.PrivilegedActionException: java.io.IOException: Service
connectivity error.
at
org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:385)
at
org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter.doFilter(ShiroSubjectIdentityAdapter.java:74)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
at
org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
at
org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at
org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at
org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at
org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at
org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at
org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at
org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at
org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at
org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
at
org.apache.hadoop.gateway.filter.ResponseCookieFilter.doFilter(ResponseCookieFilter.java:38)
at
org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
at
org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:128)
at
org.apache.hadoop.gateway.GatewayServlet.service(GatewayServlet.java:117)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:651)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:701)
Caused by: java.security.PrivilegedActionException: java.io.IOException:
Service connectivity error.
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:126)
at
org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:77)
at
org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at
org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at
org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
... 48 more
Caused by: java.io.IOException: Service connectivity error.
at
org.apache.hadoop.gateway.dispatch.HttpClientDispatch.executeRequest(HttpClientDispatch.java:126)
at
org.apache.hadoop.gateway.dispatch.HttpClientDispatch.doPut(HttpClientDispatch.java:273)
at
org.apache.hadoop.gateway.dispatch.AbstractGatewayDispatch$PutAdapter.doMethod(AbstractGatewayDispatch.java:148)
at
org.apache.hadoop.gateway.dispatch.AbstractGatewayDispatch.doFilter(AbstractGatewayDispatch.java:64)
at
org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
at
org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter.doFilter(UrlRewriteServletFilter.java:60)
at
org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
at
org.apache.hadoop.gateway.filter.security.AbstractIdentityAssertionFilter.doFilterInternal(AbstractIdentityAssertionFilter.java:209)
at
org.apache.hadoop.gateway.filter.security.AbstractIdentityAssertionFilter.continueChainAsPrincipal(AbstractIdentityAssertionFilter.java:157)
at
org.apache.hadoop.gateway.identityasserter.filter.IdentityAsserterFilter.doFilter(IdentityAsserterFilter.java:55)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212)
at
org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:93)
at
org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:90)
... 55 more
{panel}
I examined the code and discovered it is related to capped replay buffer. When
Hadoop cluster is secured and delegation token does not exist, KNOX tries to
buffer the current request for later replay in case Hadoop cluster challenges
with WWW-Authenticate
([KNOX-174|https://issues.apache.org/jira/browse/KNOX-174]) The default buffer
size is 4K. When I commented out that part, I could upload 10GB file without
problem.
I'm not sure whether this is a bug. If it's not a bug, how can I upload big
file?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
