Sorry, bad example. I send ws://<host>:<port>/gateway/pnda/pndaconsole/ socket.io/?EIO=3&transport=websocket
Knox sends /socket.io/ That yields a 400 error, Knox throws org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch protocols On Mon, Jun 18, 2018 at 6:49 PM, T Smith <ailurop...@gmail.com> wrote: > Hi Sandeep, > > Back to fighting with this. Through some nginx debugging on my backend > I've come to the conclusion that Knox isn't sending the query parameters on > to the backend, regardless of what rewrite rules I specify. > > I.e. I send /gateway/pnda/pndaconsole/socket.io/?EIO=3& > transport=polling&t=MDGlYhd > <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd> > > Knox sends /socket.io/ > <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd> > > This yields a 400 error, and Knox drops the websocket to the backend. What > was confusing me was the 101 in the browser, but I see now that Knox is > terminating the connection with the browser on one side and opening another > connection with the backend on the other. > > I see this https://git-wip-us.apache.org/repos/asf?p=knox.git;h=98a08fc > but I'm wondering if this code needs further work to support query > parameters? What do you think? > > Thanks in advance! > > /ailuropod4 > > On Thu, May 17, 2018 at 8:57 AM, T Smith <ailurop...@gmail.com> wrote: > >> Hi Sandeep, >> >> Looking at my nginx server, it never sees the transport=websocket ws:// >> protocol request, but it does see a http request for / at the corresponding >> time, so I think perhaps the root problem here is in that rewrite, as you >> mention. By the way, the reason for ws:// and not wss:// is I switched off >> TLS to remove another potential source of issue, so it's all plain http. I >> saw the same behaviour over https. >> >> My services, now, look like this - >> >> <route path="/pndaconsole/socket.io"> >> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/socket" >> to="request.url"/> >> </route> >> >> <route path="/pndaconsole/metrics"> >> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/metrics" >> to="request.url"/> >> </route> >> >> <route path="/pndaconsole/"> >> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/root" >> to="request.url"/> >> </route> >> >> <route path="/pndaconsole/**"> >> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/path" >> to="request.url"/> >> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app" >> to="response.body"/> >> </route> >> >> My inbound rewrites, now, look like this - >> >> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root" >> pattern="*://*:*/**/pndaconsole/"> >> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/> >> </rule> >> >> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket" >> pattern="*://*:*/**/pndaconsole/socket.io/?{**} >> <http://socket.io/?%7B**%7D>"> >> <rewrite template="{$serviceUrl[PNDACONSOLE]}/socket.io/?{**} >> <http://socket.io/?%7B**%7D>"/> >> </rule> >> >> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path" >> pattern="*://*:*/**/pndaconsole/{**}"> >> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/> >> </rule> >> >> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/metrics" >> pattern="*://*:*/**/pndaconsole/metrics/?{**}"> >> <rewrite template="{$serviceUrl[PNDACONSOLE]}/metrics/?{**}"/> >> </rule> >> >> The ws://...transport=websocket requests are rewritten as /, so I tried >> adding an entry in my topology for WEBSOCKET and an additional inbound rule >> like this - >> >> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket" >> pattern="ws://*:*/**/pndaconsole/socket.io/?{**} >> <http://socket.io/?%7B**%7D>"> >> <rewrite template="{$serviceUrl[WEBSSOCKET]}/socket.io/?{**} >> <http://socket.io/?%7B**%7D>"/> >> </rule> >> >> Where the topology entry is the same as PNDACONSOLE except it uses the >> ws:// protocol specifier. This didn't work at all, and in fact at that >> point, nothing got through and everything fell through to the default >> rewrite rule. >> >> I've attached the gateway log with debug turned on everywhere (tarballed >> as it's huge). >> >> I guess what I'm missing is a clear idea of what I'm supposed to do in >> the service/rewrite rules to deal with the websockets request which will >> arrive with a different protocol specifier and a different query parameter. >> Should I explicitly map those through to the back end, or is this something >> Knox handles internally? >> >> Any insight into this appreciated, it seems very close to working, and in >> fact socket.io does it's usual fallback so it is functional after a >> fashion but it seems like this should fully work given what I see with >> Zeppelin. >> >> >> Cheers, >> /ailuropod4 >> >> >> On Sun, May 13, 2018 at 3:22 PM, Sandeep Moré <moresand...@gmail.com> >> wrote: >> >>> Hello ailuropod4 >>> >>> You should not have to reroute to a different host port, this should >>> work. >>> >>> Looking at the responses, it looks like protocol switching happens at >>> the Knox end, between browser and Knox (going by the url ws:// >>> 34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/? >>> EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW) but it should have >>> been wss:// wonder why this is ws://. >>> >>> Also, can you include seperate "ws" rules for your service, similar to >>> Zeppelin. >>> Also, make sure you have, inbound routes in service.xml for e.g. >>> >>> <routes> >>> <route path="/zeppelin/ws"> >>> <rewrite apply="ZEPPELINWS/zeppelin/ws/inbound" to="request.url"/> >>> </route> >>> >>> <route path="/zeppelin/ws**"> >>> <rewrite apply="ZEPPELINWS/zeppelin/inbound" to="request.url"/> >>> </route> >>> </routes> >>> >>> and then reference it in rewrite.xml. >>> >>> Looking at the error, it looks like the issue is not with rewrite rules >>> but with the connection between Knox and the backend. >>> We can see that Knox is initiating an upgrade request but then for some >>> reason the backend service is closing/refusing it. >>> Can you turn up the logging in Knox to Debug and share it if you can >>> (this should atleast tell us whether rewrite rules are working). >>> >>> Also, do you see any errors at the backend service ? it woud be good to >>> know what the backend is seeing and reasons for it to refuse the upgrade >>> and also the error code >>> Unfortunately the error code is gobbled up somewhere in the process. >>> >>> Best, >>> Sandeep >>> >>> On Sat, May 12, 2018 at 3:16 AM, T Smith <ailurop...@gmail.com> wrote: >>> >>>> And if it helps, the stack trace as a result of this from Knox follows. >>>> >>>> Can I match a url containing &transport=websocket in one of the query >>>> parameters and map it through to a service defined as ws:// in the >>>> topology? At the moment, as above, everything is sent to a http:// >>>> service, maybe this is the crux of the problem? >>>> >>>> 2018-05-12 07:09:47,361 ERROR gateway.websockets >>>> (ProxyWebSocketAdapter.java:cleanupOnError(171)) - Error: >>>> org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch >>>> protocols >>>> 2018-05-12 07:09:47,362 ERROR gateway.websockets >>>> (ProxyWebSocketAdapter.java:onWebSocketConnect(105)) - Unable to >>>> connect to websocket server: java.io.IOException: Connect failure >>>> java.io.IOException: Connect failure >>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.connect(C >>>> lientContainer.java:157) >>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.connectTo >>>> Server(ClientContainer.java:180) >>>> at org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onW >>>> ebSocketConnect(ProxyWebSocketAdapter.java:97) >>>> at org.eclipse.jetty.websocket.common.events.JettyListenerEvent >>>> Driver.onConnect(JettyListenerEventDriver.java:87) >>>> at org.eclipse.jetty.websocket.common.events.AbstractEventDrive >>>> r.openSession(AbstractEventDriver.java:227) >>>> at org.eclipse.jetty.websocket.common.WebSocketSession.open(Web >>>> SocketSession.java:421) >>>> at org.eclipse.jetty.websocket.server.WebSocketServerConnection >>>> .onOpen(WebSocketServerConnection.java:72) >>>> at org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoi >>>> nt.java:185) >>>> at org.eclipse.jetty.server.HttpConnection.completed(HttpConnec >>>> tion.java:345) >>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java >>>> :436) >>>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConne >>>> ction.java:257) >>>> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnec >>>> tion.java:544) >>>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued >>>> ThreadPool.java:635) >>>> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedT >>>> hreadPool.java:555) >>>> at java.lang.Thread.run(Thread.java:748) >>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't >>>> switch protocols >>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.vali >>>> dateResponse(UpgradeConnection.java:314) >>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.read >>>> (UpgradeConnection.java:241) >>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.onFi >>>> llable(UpgradeConnection.java:163) >>>> ... 4 more >>>> 2018-05-12 07:09:47,373 WARN websockets.ProxyWebSocketAdapter >>>> (AbstractEventDriver.java:unhandled(245)) - Unhandled Error (closing >>>> connection) >>>> org.eclipse.jetty.io.RuntimeIOException: java.io.IOException: Connect >>>> failure >>>> at org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onW >>>> ebSocketConnect(ProxyWebSocketAdapter.java:106) >>>> at org.eclipse.jetty.websocket.common.events.JettyListenerEvent >>>> Driver.onConnect(JettyListenerEventDriver.java:87) >>>> at org.eclipse.jetty.websocket.common.events.AbstractEventDrive >>>> r.openSession(AbstractEventDriver.java:227) >>>> at org.eclipse.jetty.websocket.common.WebSocketSession.open(Web >>>> SocketSession.java:421) >>>> at org.eclipse.jetty.websocket.server.WebSocketServerConnection >>>> .onOpen(WebSocketServerConnection.java:72) >>>> at org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoi >>>> nt.java:185) >>>> at org.eclipse.jetty.server.HttpConnection.completed(HttpConnec >>>> tion.java:345) >>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java >>>> :436) >>>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConne >>>> ction.java:257) >>>> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnec >>>> tion.java:544) >>>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued >>>> ThreadPool.java:635) >>>> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedT >>>> hreadPool.java:555) >>>> at java.lang.Thread.run(Thread.java:748) >>>> Caused by: java.io.IOException: Connect failure >>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.connect(C >>>> lientContainer.java:157) >>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.connectTo >>>> Server(ClientContainer.java:180) >>>> at org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onW >>>> ebSocketConnect(ProxyWebSocketAdapter.java:97) >>>> ... 12 more >>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't >>>> switch protocols >>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.vali >>>> dateResponse(UpgradeConnection.java:314) >>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.read >>>> (UpgradeConnection.java:241) >>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.onFi >>>> llable(UpgradeConnection.java:163) >>>> ... 4 more >>>> >>>> >>>> On Fri, May 11, 2018 at 9:04 PM, T Smith <ailurop...@gmail.com> wrote: >>>> >>>>> Sorry, the other question was version. I'm using 1.0.0. >>>>> >>>>> On Fri, May 11, 2018 at 9:03 PM, T Smith <ailurop...@gmail.com> wrote: >>>>> >>>>>> Hi Sandeep, >>>>>> >>>>>> Here's what's happening - >>>>>> >>>>>> >>>>>> 1. Request URL: >>>>>> http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io >>>>>> /?EIO=3&transport=polling&t=MDGlYhd >>>>>> >>>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd> >>>>>> 2. Request Method: >>>>>> GET >>>>>> 3. Status Code: >>>>>> 200 OK >>>>>> >>>>>> >>>>>> 1. Request URL: >>>>>> ws://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/? >>>>>> EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW >>>>>> >>>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW> >>>>>> 2. Request Method: >>>>>> GET >>>>>> 3. Status Code: >>>>>> 101 Switching Protocols >>>>>> >>>>>> So far, so good. But, the next couple of requests fail, with 400 or >>>>>> 500 range errors, and the gateway log complains about protocol errors. I >>>>>> can supply these if you need them, I'm trying to be concise here to get >>>>>> the >>>>>> big picture across. >>>>>> >>>>>> Looking at tcpdump, what's happening is that ws:// request is being >>>>>> nerfed to a GET /, which on my backend happens to resolve to a nice front >>>>>> page of HTML. Websockets obviously panics as this isn't what it expected, >>>>>> and then the whole process repeats. On my backend, /socket.io is >>>>>> routed to the socket endpoint, whereas / isn't. So, my theory is that if >>>>>> I >>>>>> can somehow convince Knox to send out the ws:// request with the correct >>>>>> path including /socket.io, it might work. >>>>>> >>>>>> At the moment my service/rewrite sections are very simple, I pick up >>>>>> requests with socket.io and send them off to the backend. >>>>>> >>>>>> services.xml - >>>>>> >>>>>> <policies> >>>>>> <policy role="webappsec"/> >>>>>> <policy role="authentication" name="Anonymous"/> >>>>>> <policy role="rewrite"/> >>>>>> <policy role="authorization"/> >>>>>> </policies> >>>>>> <routes> >>>>>> <route path="/pndaconsole"> >>>>>> </route> >>>>>> <route path="/pndaconsole/**"> >>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app" >>>>>> to="response.body"/> >>>>>> </route> >>>>>> </routes> >>>>>> >>>>>> rewrite.xml - >>>>>> >>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root" >>>>>> pattern="*://*:*/**/pndaconsole/"> >>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/> >>>>>> </rule> >>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket" >>>>>> pattern="*://*:*/**/pndaconsole/socket.io/?{**} >>>>>> <http://socket.io/?%7B**%7D>"> >>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/socket.io/?{**} >>>>>> <http://socket.io/?%7B**%7D>"/> >>>>>> </rule> >>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path" >>>>>> pattern="*://*:*/**/pndaconsole/{**}"> >>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/> >>>>>> </rule> >>>>>> (I've snipped the rest of the rules handling outbound rewrites as I >>>>>> don't believe they're relevant, but let me know if you think otherwise) >>>>>> >>>>>> By all accounts it looks like it's *nearly* working, but I'm stumped >>>>>> on where my /socket.io path is going. I was considering routing all >>>>>> websockets stuff to a completely separate backend/port so that it would >>>>>> go >>>>>> to the right place regardless of how it was rewritten, but that seems a >>>>>> bit >>>>>> extreme and I was hoping to use this approach. >>>>>> >>>>>> My end to end is - >>>>>> >>>>>> client, an angularjs app --> knox --> nginx --> reverse proxied >>>>>> websockets backend in nodejs >>>>>> >>>>>> --> flat files, being an angularjs app >>>>>> >>>>>> We don't have to use socket.io, I guess, but it is currently used >>>>>> everywhere so I'd rather avoid swapping that out to get this working if >>>>>> possible. It's a fairly common library. >>>>>> >>>>>> Cheers, >>>>>> >>>>>> /ailuropod4 <ailurop...@gmail.com> >>>>>> >>>>>> >>>>>> >>>>>> On Fri, May 11, 2018 at 8:18 PM, Sandeep Moré <moresand...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I am not very familiar with socket.io apps, so you might have to >>>>>>> fill me in about what you are trying to do. >>>>>>> Looks like you are having issue rewriting Websocket url. >>>>>>> >>>>>>> What version of Knox are you using ? >>>>>>> Knox 0.13.0 and up supports better URL rewriting, see KNOX-776 >>>>>>> <https://issues.apache.org/jira/browse/KNOX-776>, it also has some >>>>>>> examples in the comments. >>>>>>> >>>>>>> What is the ws:// url you are trying to rewrite and the rules you >>>>>>> are using ? >>>>>>> >>>>>>> Best, >>>>>>> Sandeep >>>>>>> >>>>>>> On Fri, May 11, 2018 at 2:54 PM, T Smith <ailurop...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I'm struggling to get a simple socket.io based application working >>>>>>>> through Knox. >>>>>>>> >>>>>>>> I see websockets is supported, but the upgrade handshake seems to >>>>>>>> be nerfed on the way out, specifically the /socket.io/? etc part >>>>>>>> simply becomes /. >>>>>>>> >>>>>>>> I've tried lots of permutations - does anyone have a working >>>>>>>> example they can point me towards? I've looked at the Zeppelin >>>>>>>> approach, >>>>>>>> but this isn't quite the same thing (not socket.io). >>>>>>>> >>>>>>>> Any help appreciated! >>>>>>>> >>>>>>>> /ailuropod4 <ailurop...@gmail.com> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
