[ https://issues.apache.org/jira/browse/KNOX-475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Larry McCay resolved KNOX-475. ------------------------------ Resolution: Not a Problem Closing as not a problem. With the topology configured to use webhdfs rather than httpfs the upload works. > "Hit replay buffer max limit" when uploading greater than 4KB file to HDFS > -------------------------------------------------------------------------- > > Key: KNOX-475 > URL: https://issues.apache.org/jira/browse/KNOX-475 > Project: Apache Knox > Issue Type: Bug > Components: Server > Affects Versions: 0.4.0 > Environment: CentOS 6.5 > HttpFS 2.4.0, Kerberized > Oracle JDK 1.7.0_45 64bit > Reporter: jaehoon ko > > My cluster has a KNOX gateway that forwards WebHDFS REST API to HttpFS. > HttpFS and HDFS are Kerberized. > If I try uploading a file bigger than 4K, I get 500 error. > {code} > $ curl -u cersei:cersei -ik -X PUT > "https://node-02-04:8443/gateway/gc/webhdfs/v1/user/cersei/bigfile?op=create" > HTTP/1.1 307 Temporary Redirect > Set-Cookie: > JSESSIONID=wlsp2k09p1mj15i0n1tfzdai0;Path=/gateway/gc;Secure;HttpOnly > Expires: Thu, 01 Jan 1970 00:00:00 GMT > Server: Apache-Coyote/1.1 > Location: > https://node-02-04:8443/gateway/gc/webhdfs/data/v1/webhdfs/v1/user/cersei/bigfile?_=AAAACAAAABAAAABQ0N5SioA7O6sdYwm-BuvUtz3Ov6BvONMosqdDZf865RRUEsOmLDbvjh6vwMCr9kRebZ-_98ksmWnbOQkEMDbbZzYD_vrSCLTbGS9fkz1OLSqd4dOUaoIjZDsXj9QDgdbwOa6BPsjSUmg > Date: Tue, 02 Dec 2014 11:01:26 GMT > Content-Type: application/json > Content-Length: 0 > curl -ik -H "Content-Type:application/octet-stream" -u cersei:cersei -T > bigfile -X PUT > "https://node-02-04:8443/gateway/gc/webhdfs/data/v1/webhdfs/v1/user/cersei/bigfile?_=AAAACAAAABAAAABQ0N5SioA7O6sdYwm-BuvUtz3Ov6BvONMosqdDZf865RRUEsOmLDbvjh6vwMCr9kRebZ-_98ksmWnbOQkEMDbbZzYD_vrSCLTbGS9fkz1OLSqd4dOUaoIjZDsXj9QDgdbwOa6BPsjSUmg" > HTTP/1.1 100 Continue > HTTP/1.1 500 Server Error > Set-Cookie: > JSESSIONID=1t8dm6cn9rlw31mzureff6nqum;Path=/gateway/gc;Secure;HttpOnly > Content-Length: 0 > Server: Jetty(8.1.14.v20131031) > {code} > Knox log is: > {panel} > 2014-12-02 20:01:26,411 INFO realm.AuthorizingRealm > (AuthorizingRealm.java:getAuthorizationCacheLazy(248)) - No cache or > cacheManager properties have been set. Authorization cache cannot be > obtained. > 2014-12-02 20:02:08,094 INFO realm.AuthorizingRealm > (AuthorizingRealm.java:getAuthorizationCacheLazy(248)) - No cache or > cacheManager properties have been set. Authorization cache cannot be > obtained. > 2014-12-02 20:02:08,395 INFO client.DefaultHttpClient > (DefaultRequestDirector.java:tryExecute(726)) - I/O exception > (java.io.IOException) caught when processing request: Hit replay buffer max > limit > 2014-12-02 20:02:08,395 INFO client.DefaultHttpClient > (DefaultRequestDirector.java:tryExecute(733)) - Retrying request > 2014-12-02 20:02:08,405 INFO client.DefaultHttpClient > (DefaultRequestDirector.java:tryExecute(726)) - I/O exception > (java.io.IOException) caught when processing request: Hit replay buffer max > limit > 2014-12-02 20:02:08,407 INFO client.DefaultHttpClient > (DefaultRequestDirector.java:tryExecute(733)) - Retrying request > 2014-12-02 20:02:08,415 INFO client.DefaultHttpClient > (DefaultRequestDirector.java:tryExecute(726)) - I/O exception > (java.io.IOException) caught when processing request: Hit replay buffer max > limit > 2014-12-02 20:02:08,416 INFO client.DefaultHttpClient > (DefaultRequestDirector.java:tryExecute(733)) - Retrying request > 2014-12-02 20:02:08,426 WARN hadoop.gateway > (HttpClientDispatch.java:executeRequest(124)) - Connection exception > dispatching request: > http://node-02-04.gc.net:14000/webhdfs/v1/user/cersei/bigfile?op=CREATE&data=true&doAs=cersei > java.io.IOException: Hit replay buffer max limit > java.io.IOException: Hit replay buffer max limit > at > org.apache.hadoop.gateway.dispatch.CappedBufferHttpEntity$ReplayStream.read(CappedBufferHttpEntity.java:143) > at java.io.InputStream.read(InputStream.java:101) > at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792) > at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769) > at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744) > at > org.apache.hadoop.gateway.dispatch.CappedBufferHttpEntity.writeTo(CappedBufferHttpEntity.java:93) > at > org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98) > at > org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108) > at > org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:122) > at > org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:271) > at > org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:197) > at > org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:257) > at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) > at > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715) > at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520) > at > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) > at > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) > at > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) > at > org.apache.hadoop.gateway.dispatch.HttpClientDispatch.executeKerberosDispatch(HttpClientDispatch.java:193) > at > org.apache.hadoop.gateway.dispatch.HttpClientDispatch.executeRequest(HttpClientDispatch.java:120) > at > org.apache.hadoop.gateway.dispatch.HttpClientDispatch.doPut(HttpClientDispatch.java:273) > at > org.apache.hadoop.gateway.dispatch.AbstractGatewayDispatch$PutAdapter.doMethod(AbstractGatewayDispatch.java:148) > at > org.apache.hadoop.gateway.dispatch.AbstractGatewayDispatch.doFilter(AbstractGatewayDispatch.java:64) > at > org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60) > at > org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311) > at > org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212) > at > org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter.doFilter(UrlRewriteServletFilter.java:60) > at > org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60) > at > org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311) > at > org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212) > at > org.apache.hadoop.gateway.filter.security.AbstractIdentityAssertionFilter.doFilterInternal(AbstractIdentityAssertionFilter.java:209) > at > org.apache.hadoop.gateway.filter.security.AbstractIdentityAssertionFilter.continueChainAsPrincipal(AbstractIdentityAssertionFilter.java:157) > at > org.apache.hadoop.gateway.identityasserter.filter.IdentityAsserterFilter.doFilter(IdentityAsserterFilter.java:55) > at > org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311) > at > org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212) > at > org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:93) > at > org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:90) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:416) > at > org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:126) > at > org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:77) > at > org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) > at > org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) > at > org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) > at > org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter.doFilter(ShiroSubjectIdentityAdapter.java:74) > at > org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311) > at > org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212) > at > org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) > at > org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) > at > org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) > at > org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) > at > org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) > at > org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) > at > org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) > at > org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) > at > org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) > at > org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) > at > org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) > at > org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) > at > org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311) > at > org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212) > at > org.apache.hadoop.gateway.filter.ResponseCookieFilter.doFilter(ResponseCookieFilter.java:38) > at > org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60) > at > org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311) > at > org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212) > at > org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:128) > at > org.apache.hadoop.gateway.GatewayServlet.service(GatewayServlet.java:117) > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > at org.eclipse.jetty.server.Server.handle(Server.java:370) > at > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) > at > org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) > at > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:651) > at > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > at > org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) > at > org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > at java.lang.Thread.run(Thread.java:701) > 2014-12-02 20:02:08,433 ERROR hadoop.gateway > (AbstractGatewayFilter.java:doFilter(62)) - Failed to execute filter: > java.io.IOException: Service connectivity error. > 2014-12-02 20:02:08,433 ERROR hadoop.gateway > (AbstractGatewayFilter.java:doFilter(62)) - Failed to execute filter: > java.io.IOException: Service connectivity error. > 2014-12-02 20:02:08,434 ERROR hadoop.gateway > (AbstractGatewayFilter.java:doFilter(65)) - Failed to execute filter: > javax.servlet.ServletException: org.apache.shiro.subject.ExecutionException: > java.security.PrivilegedActionException: java.io.IOException: Service > connectivity error. > 2014-12-02 20:02:08,434 ERROR hadoop.gateway > (GatewayFilter.java:doFilter(134)) - Failed to execute filter: > javax.servlet.ServletException: org.apache.shiro.subject.ExecutionException: > java.security.PrivilegedActionException: java.io.IOException: Service > connectivity error. > 2014-12-02 20:02:08,435 ERROR hadoop.gateway > (GatewayServlet.java:service(122)) - Failed to execute filter: > javax.servlet.ServletException: org.apache.shiro.subject.ExecutionException: > java.security.PrivilegedActionException: java.io.IOException: Service > connectivity error. > 2014-12-02 20:02:08,435 WARN servlet.ServletHandler > (ServletHandler.java:doHandle(536)) - > javax.servlet.ServletException: org.apache.shiro.subject.ExecutionException: > java.security.PrivilegedActionException: java.io.IOException: Service > connectivity error. > at > org.apache.shiro.web.servlet.AdviceFilter.cleanup(AdviceFilter.java:196) > at > org.apache.shiro.web.filter.authc.AuthenticatingFilter.cleanup(AuthenticatingFilter.java:155) > at > org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:148) > at > org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) > at > org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) > at > org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) > at > org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) > at > org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) > at > org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) > at > org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) > at > org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) > at > org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) > at > org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311) > at > org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212) > at > org.apache.hadoop.gateway.filter.ResponseCookieFilter.doFilter(ResponseCookieFilter.java:38) > at > org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60) > at > org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311) > at > org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212) > at > org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:128) > at > org.apache.hadoop.gateway.GatewayServlet.service(GatewayServlet.java:117) > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > at org.eclipse.jetty.server.Server.handle(Server.java:370) > at > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) > at > org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) > at > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:651) > at > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > at > org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) > at > org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > at java.lang.Thread.run(Thread.java:701) > Caused by: org.apache.shiro.subject.ExecutionException: > java.security.PrivilegedActionException: java.io.IOException: Service > connectivity error. > at > org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:385) > at > org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter.doFilter(ShiroSubjectIdentityAdapter.java:74) > at > org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311) > at > org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212) > at > org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) > at > org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) > at > org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) > ... 42 more > Caused by: java.security.PrivilegedActionException: java.io.IOException: > Service connectivity error. > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:416) > at > org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:126) > at > org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:77) > at > org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) > at > org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) > at > org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) > ... 48 more > Caused by: java.io.IOException: Service connectivity error. > at > org.apache.hadoop.gateway.dispatch.HttpClientDispatch.executeRequest(HttpClientDispatch.java:126) > at > org.apache.hadoop.gateway.dispatch.HttpClientDispatch.doPut(HttpClientDispatch.java:273) > at > org.apache.hadoop.gateway.dispatch.AbstractGatewayDispatch$PutAdapter.doMethod(AbstractGatewayDispatch.java:148) > at > org.apache.hadoop.gateway.dispatch.AbstractGatewayDispatch.doFilter(AbstractGatewayDispatch.java:64) > at > org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60) > at > org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311) > at > org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212) > at > org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter.doFilter(UrlRewriteServletFilter.java:60) > at > org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60) > at > org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311) > at > org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212) > at > org.apache.hadoop.gateway.filter.security.AbstractIdentityAssertionFilter.doFilterInternal(AbstractIdentityAssertionFilter.java:209) > at > org.apache.hadoop.gateway.filter.security.AbstractIdentityAssertionFilter.continueChainAsPrincipal(AbstractIdentityAssertionFilter.java:157) > at > org.apache.hadoop.gateway.identityasserter.filter.IdentityAsserterFilter.doFilter(IdentityAsserterFilter.java:55) > at > org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311) > at > org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212) > at > org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:93) > at > org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:90) > ... 55 more > 2014-12-02 20:02:08,451 WARN servlet.ServletHandler > (ServletHandler.java:doHandle(561)) - > /gateway/gc/webhdfs/data/v1/webhdfs/v1/user/cersei/bigfile > org.apache.shiro.subject.ExecutionException: > java.security.PrivilegedActionException: java.io.IOException: Service > connectivity error. > at > org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:385) > at > org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter.doFilter(ShiroSubjectIdentityAdapter.java:74) > at > org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311) > at > org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212) > at > org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) > at > org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) > at > org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) > at > org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) > at > org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) > at > org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) > at > org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) > at > org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) > at > org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) > at > org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) > at > org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) > at > org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) > at > org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311) > at > org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212) > at > org.apache.hadoop.gateway.filter.ResponseCookieFilter.doFilter(ResponseCookieFilter.java:38) > at > org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60) > at > org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311) > at > org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212) > at > org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:128) > at > org.apache.hadoop.gateway.GatewayServlet.service(GatewayServlet.java:117) > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > at org.eclipse.jetty.server.Server.handle(Server.java:370) > at > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) > at > org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) > at > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:651) > at > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > at > org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) > at > org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > at java.lang.Thread.run(Thread.java:701) > Caused by: java.security.PrivilegedActionException: java.io.IOException: > Service connectivity error. > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:416) > at > org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:126) > at > org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:77) > at > org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) > at > org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) > at > org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) > ... 48 more > Caused by: java.io.IOException: Service connectivity error. > at > org.apache.hadoop.gateway.dispatch.HttpClientDispatch.executeRequest(HttpClientDispatch.java:126) > at > org.apache.hadoop.gateway.dispatch.HttpClientDispatch.doPut(HttpClientDispatch.java:273) > at > org.apache.hadoop.gateway.dispatch.AbstractGatewayDispatch$PutAdapter.doMethod(AbstractGatewayDispatch.java:148) > at > org.apache.hadoop.gateway.dispatch.AbstractGatewayDispatch.doFilter(AbstractGatewayDispatch.java:64) > at > org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60) > at > org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311) > at > org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212) > at > org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter.doFilter(UrlRewriteServletFilter.java:60) > at > org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:60) > at > org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311) > at > org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212) > at > org.apache.hadoop.gateway.filter.security.AbstractIdentityAssertionFilter.doFilterInternal(AbstractIdentityAssertionFilter.java:209) > at > org.apache.hadoop.gateway.filter.security.AbstractIdentityAssertionFilter.continueChainAsPrincipal(AbstractIdentityAssertionFilter.java:157) > at > org.apache.hadoop.gateway.identityasserter.filter.IdentityAsserterFilter.doFilter(IdentityAsserterFilter.java:55) > at > org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:311) > at > org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:212) > at > org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:93) > at > org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:90) > ... 55 more > {panel} > I examined the code and discovered it is related to capped replay buffer. > When Hadoop cluster is secured and delegation token does not exist, KNOX > tries to buffer the current request for later replay in case Hadoop cluster > challenges with WWW-Authenticate > ([KNOX-174|https://issues.apache.org/jira/browse/KNOX-174]) The default > buffer size is 4K. When I commented out that part, I could upload 10GB file > without problem. > I'm not sure whether this is a bug. If it's not a bug, how can I upload big > file? -- This message was sent by Atlassian JIRA (v6.3.4#6332)