[jira] [Commented] (KNOX-1997) For websocket connections, seeing java.lang.NullPointerException when a data frame arrives on websocket while connection is still being setup
[
https://issues.apache.org/jira/browse/KNOX-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16974284#comment-16974284
]
ASF subversion and git services commented on KNOX-1997:
---
Commit b144016b629be5fd4c526e25e5d40d2c4265c761 in knox's branch
refs/heads/master from Kevin Risden
[ https://gitbox.apache.org/repos/asf?p=knox.git;h=b144016 ]
KNOX-1997 - Fix diamond operator compiliation error
Signed-off-by: Kevin Risden
> For websocket connections, seeing java.lang.NullPointerException when a data
> frame arrives on websocket while connection is still being setup
> -
>
> Key: KNOX-1997
> URL: https://issues.apache.org/jira/browse/KNOX-1997
> Project: Apache Knox
> Issue Type: Bug
> Components: Server
>Affects Versions: 1.0.0, 1.1.0, 1.2.0, 1.3.0
> Environment: HDP 3.1.0
>Reporter: Rajat Goel
>Assignee: Rajat Goel
>Priority: Major
> Fix For: 1.4.0
>
> Attachments: KNOX-1997.patch, gateway.log_26Aug.gz
>
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> I am seeing following exception in Knox gateway logs:
> {code:java}
> 2019-08-23 12:43:46,347 WARN websockets.ProxyInboundClient
> (AbstractEventDriver.java:unhandled(251)) - Unhandled Error (closing
> connection)_
> java.lang.NullPointerException
> at
> org.apache.knox.gateway.websockets.ProxyWebSocketAdapter$3.onMessageText(ProxyWebSocketAdapter.java:260)
> at
> org.apache.knox.gateway.websockets.ProxyInboundClient$2.onMessage(ProxyInboundClient.java:87)
> at
> org.apache.knox.gateway.websockets.ProxyInboundClient$2.onMessage(ProxyInboundClient.java:78)
> at
> org.eclipse.jetty.websocket.jsr356.messages.TextWholeMessage.messageComplete(TextWholeMessage.java:56)
> at
> org.eclipse.jetty.websocket.jsr356.endpoints.JsrEndpointEventDriver.onTextFrame(JsrEndpointEventDriver.java:218)
> at
> org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:162)
> at
> org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:476)
> at
> org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:220)
> at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:219)
> at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:244)
> at
> org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onConnectionStateChange(AbstractWebSocketConnection.java:337)
> at
> org.eclipse.jetty.websocket.common.io.IOState.notifyStateListeners(IOState.java:184)
> at org.eclipse.jetty.websocket.common.io.IOState.onOpened(IOState.java:447)
> at
> org.eclipse.jetty.websocket.common.WebSocketSession.open(WebSocketSession.java:619)
> at
> org.eclipse.jetty.websocket.common.WebSocketSession.onOpened(WebSocketSession.java:544)
> at
> org.eclipse.jetty.io.AbstractConnection.onOpened(AbstractConnection.java:209)
> at
> org.eclipse.jetty.io.AbstractConnection.onOpen(AbstractConnection.java:202)
> at
> org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onOpen(AbstractWebSocketConnection.java:446)
> at org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoint.java:440)
> at
> org.eclipse.jetty.websocket.client.WebSocketUpgradeRequest.upgrade(WebSocketUpgradeRequest.java:624)
> at
> org.eclipse.jetty.client.http.HttpChannelOverHTTP.exchangeTerminating(HttpChannelOverHTTP.java:118)
> at
> org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:462)
> at
> org.eclipse.jetty.client.HttpReceiver.responseSuccess(HttpReceiver.java:416)
> {code}
> I added more debug logs in Knox to analyze the issue and observed following
> was happening:
> * Knox generates proper URL for backend websocket connection
> {code:java}
> 2019-08-23 12:43:46,068 DEBUG gateway.websockets
> (GatewayWebsocketHandler.java:createWebSocket(148)) - Message: Rajat ..
> backednUrl =
> ws://rafd001-mst-01.cloud.in.guavus.com:11011/_sock/411/anq2kfzp/websocket{code}
> * Websocket upgrade successful at HTTP protocol layer. Knox starts setting
> up its session management internal data structures for this connection
> {code:java}
> 2019-08-23 12:43:46,313 DEBUG client.HttpReceiver
> (HttpReceiver.java:responseSuccess(403)) - Response success
> HttpResponse[HTTP/1.1 101 Switching Protocols]@708f1c60
> _2019-08-23 12:43:46,324 DEBUG common.WebSocketSession
> (WebSocketSession.java:doStart(248)) - starting -
> WebSocketSession[websocket=JsrEndpointEventDriver[org.apache.knox.gateway.websockets.ProxyInboundClient],behavior=CLIENT,connection=WebSocketClientConnection@1528d316::SocketChannelEndPoint@10d45033
> {[rafd001-mst-01.cloud.in.gua
[jira] [Commented] (KNOX-2055) Document responseExcludeHeaders in ConfigurableDispatch
[
https://issues.apache.org/jira/browse/KNOX-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16974316#comment-16974316
]
Kevin Risden commented on KNOX-2055:
[~smolnar] I don't think any of the dispatch details are in the user guide. I
think most of that detail is in the dev guide.
https://knox.apache.org/books/knox-1-3-0/dev-guide.html
This looks like a great addition to the dev guide.
> Document responseExcludeHeaders in ConfigurableDispatch
> ---
>
> Key: KNOX-2055
> URL: https://issues.apache.org/jira/browse/KNOX-2055
> Project: Apache Knox
> Issue Type: Task
> Components: Site
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Sandor Molnar
>Assignee: Sandor Molnar
>Priority: Minor
> Fix For: 1.4.0
>
> Attachments: KNOX-2055.patch, Screen Shot 2019-11-13 at 11.56.29
> AM.png
>
>
> As of now, there is no documentation on the {{responseExcludeHeaders}}
> parameter neither in the user guide nor in the developer guide.
> With a recent change, the behavior of this feature has been changed so that
> it's time to document it somewhere.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KNOX-2125) Zeppelin 0.8.2 mapping error
Daryl Erwin created KNOX-2125: - Summary: Zeppelin 0.8.2 mapping error Key: KNOX-2125 URL: https://issues.apache.org/jira/browse/KNOX-2125 Project: Apache Knox Issue Type: Bug Components: KnoxSSO Affects Versions: 1.3.0 Reporter: Daryl Erwin Getting this error .. and it seems to mean it wont show the editor. 2019-11-14 15:49:10,442 WARN knox.gateway (GatewayFilter.java:doFilter(181)) - Failed to match path /zeppelin/app.223f1250e5a93cdb5cdb.js.map 2019-11-14 15:49:10,564 WARN knox.gateway (GatewayFilter.java:doFilter(181)) - Failed to match path /zeppelin/app.223f1250e5a93cdb5cdb.css.map -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KNOX-2125) Zeppelin 0.8.2 mapping error
[ https://issues.apache.org/jira/browse/KNOX-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Risden updated KNOX-2125: --- Component/s: (was: KnoxSSO) > Zeppelin 0.8.2 mapping error > > > Key: KNOX-2125 > URL: https://issues.apache.org/jira/browse/KNOX-2125 > Project: Apache Knox > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Daryl Erwin >Priority: Major > > Getting this error .. and it seems to mean it wont show the editor. > 2019-11-14 15:49:10,442 WARN knox.gateway (GatewayFilter.java:doFilter(181)) > - Failed to match path /zeppelin/app.223f1250e5a93cdb5cdb.js.map > 2019-11-14 15:49:10,564 WARN knox.gateway (GatewayFilter.java:doFilter(181)) > - Failed to match path /zeppelin/app.223f1250e5a93cdb5cdb.css.map -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KNOX-2125) Zeppelin 0.8.2 mapping error
[ https://issues.apache.org/jira/browse/KNOX-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16974394#comment-16974394 ] Kevin Risden commented on KNOX-2125: [~allknowing2012] do you know if this is isolated to Zeppelin 0.8.2 (ie: something changed in Zeppelin) or is it something that broke in Knox 1.3.0? It looks like you are trying to proxy Zeppelin with Knox so your url is something like the following correct? https://KNOX_HOST:KNOX_PORT/gateway/TOPOLOGY/zeppelin/ > Zeppelin 0.8.2 mapping error > > > Key: KNOX-2125 > URL: https://issues.apache.org/jira/browse/KNOX-2125 > Project: Apache Knox > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Daryl Erwin >Priority: Major > > Getting this error .. and it seems to mean it wont show the editor. > 2019-11-14 15:49:10,442 WARN knox.gateway (GatewayFilter.java:doFilter(181)) > - Failed to match path /zeppelin/app.223f1250e5a93cdb5cdb.js.map > 2019-11-14 15:49:10,564 WARN knox.gateway (GatewayFilter.java:doFilter(181)) > - Failed to match path /zeppelin/app.223f1250e5a93cdb5cdb.css.map -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KNOX-2125) Zeppelin 0.8.2 mapping error
[ https://issues.apache.org/jira/browse/KNOX-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16974394#comment-16974394 ] Kevin Risden edited comment on KNOX-2125 at 11/14/19 4:13 PM: -- [~allknowing2012] do you know if this is isolated to Zeppelin 0.8.2 (ie: something changed in Zeppelin) or is it something that broke in Knox 1.3.0 (ie: this worked on older Knox versions)? It looks like you are trying to proxy Zeppelin with Knox so your url is something like the following correct? https://KNOX_HOST:KNOX_PORT/gateway/TOPOLOGY/zeppelin/ was (Author: risdenk): [~allknowing2012] do you know if this is isolated to Zeppelin 0.8.2 (ie: something changed in Zeppelin) or is it something that broke in Knox 1.3.0? It looks like you are trying to proxy Zeppelin with Knox so your url is something like the following correct? https://KNOX_HOST:KNOX_PORT/gateway/TOPOLOGY/zeppelin/ > Zeppelin 0.8.2 mapping error > > > Key: KNOX-2125 > URL: https://issues.apache.org/jira/browse/KNOX-2125 > Project: Apache Knox > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Daryl Erwin >Priority: Major > > Getting this error .. and it seems to mean it wont show the editor. > 2019-11-14 15:49:10,442 WARN knox.gateway (GatewayFilter.java:doFilter(181)) > - Failed to match path /zeppelin/app.223f1250e5a93cdb5cdb.js.map > 2019-11-14 15:49:10,564 WARN knox.gateway (GatewayFilter.java:doFilter(181)) > - Failed to match path /zeppelin/app.223f1250e5a93cdb5cdb.css.map -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KNOX-2125) Zeppelin 0.8.2 mapping error
[ https://issues.apache.org/jira/browse/KNOX-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16974490#comment-16974490 ] Daryl Erwin commented on KNOX-2125: --- Using a docker environment I tried back with 0.8.0 and .1 - same results. This is a new install so only tried with knox 1.3.0 (and Okta and SAML) Correct with the link gateway/sandbox/zeppelin/ Chrome inspect is also throwing this.. (which might be related ie .. is it defined in those missing .map files?) vendor.d3e00d53c8ffcdb4.js:37 TypeError: e.map is not a function at app.223f1250e5a93cdb5cdb.js:48 at g (vendor.d3e00d53c8ffcdb4.js:38) at vendor.d3e00d53c8ffcdb4.js:38 at o.$eval (vendor.d3e00d53c8ffcdb4.js:38) at o.$digest (vendor.d3e00d53c8ffcdb4.js:38) at o.$apply (vendor.d3e00d53c8ffcdb4.js:38) at i (vendor.d3e00d53c8ffcdb4.js:37) at u (vendor.d3e00d53c8ffcdb4.js:37) at XMLHttpRequest.x.onload (vendor.d3e00d53c8ffcdb4.js:37) > Zeppelin 0.8.2 mapping error > > > Key: KNOX-2125 > URL: https://issues.apache.org/jira/browse/KNOX-2125 > Project: Apache Knox > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Daryl Erwin >Priority: Major > > Getting this error .. and it seems to mean it wont show the editor. > 2019-11-14 15:49:10,442 WARN knox.gateway (GatewayFilter.java:doFilter(181)) > - Failed to match path /zeppelin/app.223f1250e5a93cdb5cdb.js.map > 2019-11-14 15:49:10,564 WARN knox.gateway (GatewayFilter.java:doFilter(181)) > - Failed to match path /zeppelin/app.223f1250e5a93cdb5cdb.css.map -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KNOX-2125) Zeppelin 0.8.2 mapping error
[ https://issues.apache.org/jira/browse/KNOX-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16974546#comment-16974546 ] Daryl Erwin commented on KNOX-2125: --- Can I add lines to rewrite.xml - once added do they automatically get picked up/ > Zeppelin 0.8.2 mapping error > > > Key: KNOX-2125 > URL: https://issues.apache.org/jira/browse/KNOX-2125 > Project: Apache Knox > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Daryl Erwin >Priority: Major > > Getting this error .. and it seems to mean it wont show the editor. > 2019-11-14 15:49:10,442 WARN knox.gateway (GatewayFilter.java:doFilter(181)) > - Failed to match path /zeppelin/app.223f1250e5a93cdb5cdb.js.map > 2019-11-14 15:49:10,564 WARN knox.gateway (GatewayFilter.java:doFilter(181)) > - Failed to match path /zeppelin/app.223f1250e5a93cdb5cdb.css.map -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KNOX-2125) Zeppelin 0.8.2 mapping error
[ https://issues.apache.org/jira/browse/KNOX-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16974584#comment-16974584 ] Daryl Erwin commented on KNOX-2125: --- Happens in 1.3.0 Knox .. WORKS in 1.2.0 :) Atleast I have a working solution .. > Zeppelin 0.8.2 mapping error > > > Key: KNOX-2125 > URL: https://issues.apache.org/jira/browse/KNOX-2125 > Project: Apache Knox > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Daryl Erwin >Priority: Major > > Getting this error .. and it seems to mean it wont show the editor. > 2019-11-14 15:49:10,442 WARN knox.gateway (GatewayFilter.java:doFilter(181)) > - Failed to match path /zeppelin/app.223f1250e5a93cdb5cdb.js.map > 2019-11-14 15:49:10,564 WARN knox.gateway (GatewayFilter.java:doFilter(181)) > - Failed to match path /zeppelin/app.223f1250e5a93cdb5cdb.css.map -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [knox] shanyu commented on a change in pull request #177: [WIP] KNOX-2095 - Adding in DefaultDispatch code and tests to handle 504 errors
shanyu commented on a change in pull request #177: [WIP] KNOX-2095 - Adding in
DefaultDispatch code and tests to handle 504 errors
URL: https://github.com/apache/knox/pull/177#discussion_r346570126
##
File path:
gateway-spi/src/main/java/org/apache/knox/gateway/dispatch/DefaultDispatch.java
##
@@ -138,6 +144,10 @@ protected HttpResponse executeOutboundRequest(
HttpUriRequest outboundRequest )
}
}
auditor.audit( Action.DISPATCH, outboundRequest.getURI().toString(),
ResourceType.URI, ActionOutcome.SUCCESS, RES.responseStatus( statusCode ) );
+} catch( SocketTimeoutException e ) {
+//Set a 504 instead of throwing an IO exception in the event of a
socket timeout,
+//as an IO exception forces a 500 to be displayed.
Review comment:
@risdenk , for active/passive, it doesn't make sense to failover from active
to passive in the case of socket timeout. For active/active, it may have value
to failover. However, I have 2 arguments against this:
1) Usually the client side that talks to knox has its own timeout (think
about a Jupyter notebook), so letting service failover in the case of socket
timeout multiple times would eventually cause client side timeout.
2) Most likely the timeout is because the request itself took too much time
exceeding the socket timeout config (considering Livy session creation), so
failover to another server would also cause a socket timeout as well.
And for active-active type of service, if really needed we can always retry
at the client side.
I think letting client side know that the request caused timeout is very
important, instead of Letting them face a generic 500 error without knowing
what to do next.
Or we could
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (KNOX-2095) Many errors (E.G. 504s) being masked as 500 errors
[
https://issues.apache.org/jira/browse/KNOX-2095?focusedWorklogId=343782&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-343782
]
ASF GitHub Bot logged work on KNOX-2095:
Author: ASF GitHub Bot
Created on: 14/Nov/19 22:04
Start Date: 14/Nov/19 22:04
Worklog Time Spent: 10m
Work Description: shanyu commented on pull request #177: [WIP] KNOX-2095
- Adding in DefaultDispatch code and tests to handle 504 errors
URL: https://github.com/apache/knox/pull/177#discussion_r346570126
##
File path:
gateway-spi/src/main/java/org/apache/knox/gateway/dispatch/DefaultDispatch.java
##
@@ -138,6 +144,10 @@ protected HttpResponse executeOutboundRequest(
HttpUriRequest outboundRequest )
}
}
auditor.audit( Action.DISPATCH, outboundRequest.getURI().toString(),
ResourceType.URI, ActionOutcome.SUCCESS, RES.responseStatus( statusCode ) );
+} catch( SocketTimeoutException e ) {
+//Set a 504 instead of throwing an IO exception in the event of a
socket timeout,
+//as an IO exception forces a 500 to be displayed.
Review comment:
@risdenk , for active/passive, it doesn't make sense to failover from active
to passive in the case of socket timeout. For active/active, it may have value
to failover. However, I have 2 arguments against this:
1) Usually the client side that talks to knox has its own timeout (think
about a Jupyter notebook), so letting service failover in the case of socket
timeout multiple times would eventually cause client side timeout.
2) Most likely the timeout is because the request itself took too much time
exceeding the socket timeout config (considering Livy session creation), so
failover to another server would also cause a socket timeout as well.
And for active-active type of service, if really needed we can always retry
at the client side.
I think letting client side know that the request caused timeout is very
important, instead of Letting them face a generic 500 error without knowing
what to do next.
Or we could
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 343782)
Remaining Estimate: 165h 50m (was: 166h)
Time Spent: 2h 10m (was: 2h)
> Many errors (E.G. 504s) being masked as 500 errors
> --
>
> Key: KNOX-2095
> URL: https://issues.apache.org/jira/browse/KNOX-2095
> Project: Apache Knox
> Issue Type: Improvement
>Affects Versions: 1.2.0, 1.3.0
>Reporter: James Chen
>Assignee: James Chen
>Priority: Minor
> Labels: easyfix
> Fix For: 1.4.0
>
> Attachments: KNOX-2095.patch, jamchen504patch.patch
>
> Original Estimate: 168h
> Time Spent: 2h 10m
> Remaining Estimate: 165h 50m
>
> When errors occur while accessing the Knox gateway, errors are forcibly
> overridden and represented as 500 errors, rather than whatever errors they
> should be.
> For example, when the timeout value under gateway.httpclient.socketTimeout is
> set to a very low timeout value (E.G. 1 ms) under gateway-site.xml, a socket
> timeout exception is produced by the getHttpClient().execute(
> outboundRequest) call. However, this is caught by the surrounding try-catch
> block and thrown again as an IOException. This results in a generic 500
> error, rather than a 504 error one would normally expect from this sort of
> interaction.
>
> For these sorts of scenarios, I believe it would be prudent to create a dummy
> HttpResponse using a HttpResponseFactory object for the inboundResponse with
> the corresponding error code (E.G. HttpStatus.SC_GATEWAY_TIMEOUT in the event
> of a SocketTimeoutException) and return that instead to trigger the
> appropriate 504 error. I suspect there are other sorts of potential error
> code triggers that get this same IOException treatment that would be better
> off receiving their own error codes.
>
> Judging from the source code, this issue likely affects version 1.3.0, though
> this has not been tested.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] KIP-14 - KnoxShell Improvement for Tabular Data
That is a fair point and this sort of processing should not be considered a primary goal of Knox. However, as a client environment for remote clusters that will be increasingly locked down from SSH, it would be good to be able to do meaningful things as appropriate. Appropriate scenarios would be for small tasks, demonstrations and dev/testing work. As far as general purpose table operations and JDBC - not all data sources are from JDBC. Indeed those that are JDBC based sources already have those capabilities but being able to join tabular data across data source types - say an oracle database and a csv file and publish to a hive table in a cluster would make sense. There is no goal here to reinvent dataframes to actually compete in that space. On Mon, Nov 11, 2019 at 2:25 PM Kevin Risden wrote: > We need to be careful about how far this work goes into data processing. > There is a difference between providing a client side DSL to interact with > cluster services and building a completely standalone data processing DSL. > > I worry that the sorting/joining/processing is straying too far from just > being a client side DSL and turning into a whole data processing framework. > Specifically UC 3 seems to be significantly outside of the scope of what > Knox has historically done. Most of that processing should be pushed down > to the engine exposed via UC 1. I don't see why the Knox DSL should provide > that type of interaction when it should be pushed down to the JDBC > interface and the results returned through Knox. > > Kevin Risden > > > On Wed, Nov 6, 2019 at 8:42 PM larry mccay wrote: > > > All - > > > > I've created the following KIP to try and capture the motivation, > usecases > > and vision for KnoxShellTable in the following KIP. > > > > I may extend this rather than add a new KIP for the Custom Groovy Shell > > commands that I am working on for working with KnoxShellTable but in the > > meantime, this represents the core work for tabular data. Some details of > > the current implementation are still missing and may be added as we go. > > > > > > > https://cwiki.apache.org/confluence/display/KNOX/KIP-14+-+KnoxShell+Improvements+for+Tabular+Data > > > > Any thoughts on this are more than welcome! > > > > thanks, > > > > --larry > > >
Re: [DISCUSS] Planning for Apache Knox 1.4
The entirety of KIP-14 doesn't need to be scoped for the 1.4 release. JIRAs do need to be filed for the work detailed there and iterative progress will be made across multiple releases. I think that the initial goal should be to fill the role that the KnoxLine example represented and drove interest in a Knox SQL client. Being able to work with the results of a SQL query is an added benefit of this approach and we can have some basic operations for doing just that. I don't see the overall Knox metadata endpoint making it in the 1.4 release timeframe. I do see some of the underlying capabilities of things like datasources being configurable and used to interact with SQL engines rather than having to provide connection details for each session. I have an early version of this already working locally. On Mon, Nov 11, 2019 at 2:38 PM Kevin Risden wrote: > Based on that we are ~1/2 way through November and Thanksgiving in the US > around the corner, I don't see much of the KnoxShell pieces getting > integrated before end of November to make an end of November release. > Specifically the following section: > > With the continued increase in cloud based deployments and Knox Gateway use > > in securely accessing the exposed resources, we will concentrate on > > KnoxShell as a first class environment for this access. This will likely > > include an API for discovering metadata about the resources exposed > through > > Knox, the required authentication mechanisms, resource types and public > > certs. It will also include Custom GroovyShell Commands for the KnoxShell > > environment to help interact with the remote clusters and resultsets as > > local in-memory tables. I will be start a KIP to try and articulate this > > vision and related 1.4. usecases as well. > > > > The KIP was just started so would be good to flesh that out more instead of > rushing it into 1.4.0. There are a lot of moving pieces in that paragraph > and would be good to make sure the Jiras are created and scoped > appropriately. > > In addition to what was mentioned as features, there have been multiple new > service definitions (Impala, Kudu, NiFi Registry) added as well as fixes to > existing service definitions (Atlas, Livy, Ranger, Spark, YARN). > > So +1 to an end of November release, but need to make sure not trying to > rush in new things just because a release will happen. There will be more > releases. > > > Kevin Risden > > > On Fri, Nov 1, 2019 at 11:51 AM Sandeep Moré > wrote: > > > Thanks for starting the planning thread Larry ! > > Agree with the theme and the release date for 1.4.0. > > > > +1 > > > > Best, > > Sandeep > > > > On Thu, Oct 31, 2019 at 11:53 AM larry mccay wrote: > > > >> Folks - > >> > >> Out last release with end of July, I apologize for the delay in starting > >> the planning thread for 1.4. > >> > >> We currently have a backlog of ~65 JIRAs slated for a Fix Version of > 1.4. > >> > >> There has been some work going on within KnoxShell to provide a general > >> purpose representation for tabular data. This will be leveraged for > >> rendering SQL query results as well as CSV files and simple processing > >> within KnoxShell. I will be writing up a KIP to represent the overall > >> vision for this work and initial set of usecases. > >> > >> We also have Cloudera Manager based discovery emerging and we should > >> target an initial set of services to enable for CM/CDH and CDP > deployments > >> where CM is available. > >> > >> With the continued increase in cloud based deployments and Knox Gateway > >> use in securely accessing the exposed resources, we will concentrate on > >> KnoxShell as a first class environment for this access. This will likely > >> include an API for discovering metadata about the resources exposed > through > >> Knox, the required authentication mechanisms, resource types and public > >> certs. It will also include Custom GroovyShell Commands for the > KnoxShell > >> environment to help interact with the remote clusters and resultsets as > >> local in-memory tables. I will be start a KIP to try and articulate this > >> vision and related 1.4. usecases as well. > >> > >> I propose that the CM based Service Discovery and KnoxShell access to > >> remote clusters be the primary themes of the Apache Knox 1.4 release. > >> > >> I also propose that we target the end of November as the release date > for > >> 1.4. > >> > >> Thoughts? > >> > >> --larry > >> > > >
[GitHub] [knox] shanyu commented on a change in pull request #177: [WIP] KNOX-2095 - Adding in DefaultDispatch code and tests to handle 504 errors
shanyu commented on a change in pull request #177: [WIP] KNOX-2095 - Adding in
DefaultDispatch code and tests to handle 504 errors
URL: https://github.com/apache/knox/pull/177#discussion_r346570126
##
File path:
gateway-spi/src/main/java/org/apache/knox/gateway/dispatch/DefaultDispatch.java
##
@@ -138,6 +144,10 @@ protected HttpResponse executeOutboundRequest(
HttpUriRequest outboundRequest )
}
}
auditor.audit( Action.DISPATCH, outboundRequest.getURI().toString(),
ResourceType.URI, ActionOutcome.SUCCESS, RES.responseStatus( statusCode ) );
+} catch( SocketTimeoutException e ) {
+//Set a 504 instead of throwing an IO exception in the event of a
socket timeout,
+//as an IO exception forces a 500 to be displayed.
Review comment:
@risdenk , for active/passive, it doesn't make sense to failover from active
to passive in the case of socket timeout. For active/active, it may have value
to failover. However, I have 2 arguments against this:
1) Usually the client side that talks to knox has its own timeout (think
about a Jupyter notebook), so letting service failover in the case of socket
timeout multiple times would eventually cause client side timeout.
2) Most likely the timeout is because the request itself took too much time
exceeding the socket timeout config (considering Livy session creation), so
failover to another server would also cause a socket timeout as well.
And for active-active type of service, if really needed we can always retry
at the client side.
I think letting client side know that the request caused timeout is very
important, instead of Letting them face a generic 500 error without knowing
what to do next.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (KNOX-2095) Many errors (E.G. 504s) being masked as 500 errors
[
https://issues.apache.org/jira/browse/KNOX-2095?focusedWorklogId=343897&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-343897
]
ASF GitHub Bot logged work on KNOX-2095:
Author: ASF GitHub Bot
Created on: 15/Nov/19 00:29
Start Date: 15/Nov/19 00:29
Worklog Time Spent: 10m
Work Description: shanyu commented on pull request #177: [WIP] KNOX-2095
- Adding in DefaultDispatch code and tests to handle 504 errors
URL: https://github.com/apache/knox/pull/177#discussion_r346570126
##
File path:
gateway-spi/src/main/java/org/apache/knox/gateway/dispatch/DefaultDispatch.java
##
@@ -138,6 +144,10 @@ protected HttpResponse executeOutboundRequest(
HttpUriRequest outboundRequest )
}
}
auditor.audit( Action.DISPATCH, outboundRequest.getURI().toString(),
ResourceType.URI, ActionOutcome.SUCCESS, RES.responseStatus( statusCode ) );
+} catch( SocketTimeoutException e ) {
+//Set a 504 instead of throwing an IO exception in the event of a
socket timeout,
+//as an IO exception forces a 500 to be displayed.
Review comment:
@risdenk , for active/passive, it doesn't make sense to failover from active
to passive in the case of socket timeout. For active/active, it may have value
to failover. However, I have 2 arguments against this:
1) Usually the client side that talks to knox has its own timeout (think
about a Jupyter notebook), so letting service failover in the case of socket
timeout multiple times would eventually cause client side timeout.
2) Most likely the timeout is because the request itself took too much time
exceeding the socket timeout config (considering Livy session creation), so
failover to another server would also cause a socket timeout as well.
And for active-active type of service, if really needed we can always retry
at the client side.
I think letting client side know that the request caused timeout is very
important, instead of Letting them face a generic 500 error without knowing
what to do next.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 343897)
Remaining Estimate: 165h 40m (was: 165h 50m)
Time Spent: 2h 20m (was: 2h 10m)
> Many errors (E.G. 504s) being masked as 500 errors
> --
>
> Key: KNOX-2095
> URL: https://issues.apache.org/jira/browse/KNOX-2095
> Project: Apache Knox
> Issue Type: Improvement
>Affects Versions: 1.2.0, 1.3.0
>Reporter: James Chen
>Assignee: James Chen
>Priority: Minor
> Labels: easyfix
> Fix For: 1.4.0
>
> Attachments: KNOX-2095.patch, jamchen504patch.patch
>
> Original Estimate: 168h
> Time Spent: 2h 20m
> Remaining Estimate: 165h 40m
>
> When errors occur while accessing the Knox gateway, errors are forcibly
> overridden and represented as 500 errors, rather than whatever errors they
> should be.
> For example, when the timeout value under gateway.httpclient.socketTimeout is
> set to a very low timeout value (E.G. 1 ms) under gateway-site.xml, a socket
> timeout exception is produced by the getHttpClient().execute(
> outboundRequest) call. However, this is caught by the surrounding try-catch
> block and thrown again as an IOException. This results in a generic 500
> error, rather than a 504 error one would normally expect from this sort of
> interaction.
>
> For these sorts of scenarios, I believe it would be prudent to create a dummy
> HttpResponse using a HttpResponseFactory object for the inboundResponse with
> the corresponding error code (E.G. HttpStatus.SC_GATEWAY_TIMEOUT in the event
> of a SocketTimeoutException) and return that instead to trigger the
> appropriate 504 error. I suspect there are other sorts of potential error
> code triggers that get this same IOException treatment that would be better
> off receiving their own error codes.
>
> Judging from the source code, this issue likely affects version 1.3.0, though
> this has not been tested.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KNOX-2126) Add an API to retrieve the public key used for Knox JWT verification
Siddharth Seth created KNOX-2126: Summary: Add an API to retrieve the public key used for Knox JWT verification Key: KNOX-2126 URL: https://issues.apache.org/jira/browse/KNOX-2126 Project: Apache Knox Issue Type: Task Reporter: Siddharth Seth It can be useful to get access to the PublicKey used by Knox to verify the tokens that it generates outside of Knox. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KNOX-2126) Add an API to retrieve the public key used for Knox JWT verification
[ https://issues.apache.org/jira/browse/KNOX-2126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Risden updated KNOX-2126: --- Issue Type: New Feature (was: Task) > Add an API to retrieve the public key used for Knox JWT verification > > > Key: KNOX-2126 > URL: https://issues.apache.org/jira/browse/KNOX-2126 > Project: Apache Knox > Issue Type: New Feature >Reporter: Siddharth Seth >Priority: Major > > It can be useful to get access to the PublicKey used by Knox to verify the > tokens that it generates outside of Knox. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [knox] sidseth opened a new pull request #195: KNOX-2126. Add an API to retrieve the public key used for Knox JWT verification.
sidseth opened a new pull request #195: KNOX-2126. Add an API to retrieve the public key used for Knox JWT verification. URL: https://github.com/apache/knox/pull/195 ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? Not tested yet. Looking for some feedback on client side changes, if any are required. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (KNOX-2126) Add an API to retrieve the public key used for Knox JWT verification
[ https://issues.apache.org/jira/browse/KNOX-2126?focusedWorklogId=343956&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-343956 ] ASF GitHub Bot logged work on KNOX-2126: Author: ASF GitHub Bot Created on: 15/Nov/19 02:46 Start Date: 15/Nov/19 02:46 Worklog Time Spent: 10m Work Description: sidseth commented on pull request #195: KNOX-2126. Add an API to retrieve the public key used for Knox JWT verification. URL: https://github.com/apache/knox/pull/195 ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? Not tested yet. Looking for some feedback on client side changes, if any are required. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 343956) Remaining Estimate: 0h Time Spent: 10m > Add an API to retrieve the public key used for Knox JWT verification > > > Key: KNOX-2126 > URL: https://issues.apache.org/jira/browse/KNOX-2126 > Project: Apache Knox > Issue Type: New Feature >Reporter: Siddharth Seth >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > It can be useful to get access to the PublicKey used by Knox to verify the > tokens that it generates outside of Knox. -- This message was sent by Atlassian Jira (v8.3.4#803005)
