Thanks Sandeep,
that makes a good deal of sense. Obviously I need those {collection=**} to
capture the collection itself etc.
Final hurdle now seems to be on renaming the service to SOLRAPI (I've no
intention to proxy the UI as this time).
New service/rewrite here:
https://github.com/mcparlandjcgi/knox/tree/KNOX-528/gateway-service-definitions/src/main/resources/services/solr/5.5.0
And the updated topology includes the following:
<service>
<role>SOLRAPI</role>
<url>http://sandbox.hortonworks.com:8983/solr</url>
</service>
But the logs, when I send a request through, show this;
2016-11-19 20:43:11,605 DEBUG hadoop.gateway
(UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL:
https://hdp24sandbox:8443/gateway/sandbox/solr/KnoxIntegrationConfig/select?q=*.*&wt=json&indent=true,
direction: IN via explicit rule: SOLRAPI/solr/inbound/query to URL:
SOLR/KnoxIntegrationConfig/select?indent=true&q=*.*&wt=json
I don't understand why the old "SOLR" is being used at the start of the
re-written URL. It's almost as if the gateway (which I restart using
bin/gateway.sh stop / bin/gateway.sh start in-between changes) isn't picking up
the new topology correctly.
Thanks,
John McParland MIET CEng | System Architect, ODSC
Health Local and Scotland | CGI
CGI Ltd (UK)
Second Floor, Inovo Building, 121 George St, Glasgow, UK, G1 1RD
M: +44 7920 183 019
john.mcparl...@cgi.com | www.cgi-group.co.uk
CGI IT UK Limited. A CGI Group Inc. Company
Registered Office: 250 Brook Drive, Green Park, Reading RG2 6UA,
United Kingdom. Registered in England & Wales - Number 947968
CONFIDENTIALITY NOTICE: Proprietary/Confidential Information belonging to CGI
Group Inc. and its affiliates may be contained in this message. If you are not
a recipient indicated or intended in this message (or responsible for delivery
of this message to such person), or you think for any reason that this message
may have been addressed to you in error, you may not use or copy or deliver
this message to anyone else. In such case, you should destroy this message and
are asked to notify the sender by reply e-mail.
________________________________________
From: Sandeep More [moresand...@gmail.com]
Sent: 19 November 2016 17:56
To: dev@knox.apache.org
Subject: Re: Re-write rules not being applied to new service
Hello John,
You have some minor errors that are causing the paths to not match they way
you want them to. I did some local testing and the following service
definitions appear to work locally on my machine ( using your github
example as baseline )
*service.xml (complete file)*
<service role="SOLR" name="solr" version="5.5.0">
<policies>
<policy role="webappsec"/>
<policy role="authentication" name="Anonymous"/>
<policy role="rewrite"/>
<policy role="authorization"/>
</policies>
<routes>
<route path="/solr/**/**?**">
<rewrite apply="SOLR/solr/inbound/query" to="request.url"/>
</route>
</routes>
</service>
*rewrite.xml (complete file)*
<rules>
<rule dir="IN" name="SOLR/solr/inbound/query"
pattern="*://*:*/**/solr/{collection=**}/{query=**}?{**}">
<rewrite
template="{$serviceUrl[SOLR]}/{collection=**}/{query=**}?{**}"/>
</rule>
</rules>
your topology file seems right. I was able to get the basic
"gettingstarted" solr query to work.
You probably need to add some rules for the UI if you like to support it.
Also, you might want to call the above service something like "SOLRAPI"
rather then "SOLR" just to differentiate if from UI traffic.
There are minor regex changes I had to do to make it work. Let me know if
it works !
To understand UI Proxying with Knox refer to this excellent guide by Sumit
Gupta -
https://cwiki.apache.org/confluence/display/KNOX/Proxying+a+UI+using+Knox
Another good resource on adding a service to Knox by Kevin Minder is - -
https://cwiki.apache.org/confluence/display/KNOX/2015/12/17/Adding+a+service+to+Apache+Knox
These blogs explain in-detail process to add services and are a step by
step guides to adding support for knox proxying.
You also asked about debugging, generally for adding UIs I just tail the
logs and try to figure out from logs and change the logging level if
needed. If you would like to debug using an IDE you can use the command "ant
start-debug-gateway" from the knox source directory.
Sorry for the lengthy email, hope it addresses your questions !
Best,
Sandeep
On Fri, Nov 18, 2016 at 4:29 PM, McParland, John <john.mcparl...@cgi.com>
wrote:
> Thanks again Sandeep,
>
> I tried your suggestion, and while I can understand it and believe it
> should have worked, it still doesn't! I have the same error message as
> below.
>
> From my own understanding, I think the rules in rewrite.xml are fine, and
> the JUnit test proves those should work.
>
> However I haven't understood how the route in service.xml is
> picked/matched with the incoming URL. And this appears (again from what I
> understand) to be where the problem is. My understanding of the processing
> of an incoming URL is;
>
> 1) Check URL against routes (not clear on this)
> 2) Find re-write rule associated with it (presume this is just a lookup
> of a map of URL -> rule name)
> 3) Check the URL matches the pattern in the re-write rule
> 4) Apply the transformation in the template of the re-write rule.
>
> Is there anything else I can do to debug this part (e.g. see what routes
> we have at runtime)?
>
> Thanks,
>
> John McParland MIET CEng | System Architect, ODSC
> Health Local and Scotland | CGI
> CGI Ltd (UK)
> Second Floor, Inovo Building, 121 George St, Glasgow, UK, G1 1RD
> M: +44 7920 183 019
> john.mcparl...@cgi.com | www.cgi-group.co.uk
>
> CGI IT UK Limited. A CGI Group Inc. Company
> Registered Office: 250 Brook Drive, Green Park, Reading RG2 6UA,
> United Kingdom. Registered in England & Wales - Number 947968
>
> CONFIDENTIALITY NOTICE: Proprietary/Confidential Information belonging to
> CGI Group Inc. and its affiliates may be contained in this message. If you
> are not a recipient indicated or intended in this message (or responsible
> for delivery of this message to such person), or you think for any reason
> that this message may have been addressed to you in error, you may not use
> or copy or deliver this message to anyone else. In such case, you should
> destroy this message and are asked to notify the sender by reply e-mail.
>
> ________________________________________
> From: Sandeep More [moresand...@gmail.com]
> Sent: 18 November 2016 16:37
> To: dev@knox.apache.org
> Subject: Re: Re-write rules not being applied to new service
>
> Ah ! got it now and the pattern seems perfectly fine.
> In which case I think just the following should work (without any
> additional rules):
>
> service.xml
>
> <routes>
> <route path="/solr/**/**?{**}" >
> <rewrite apply="SOLR/solr/inbound/query"
> to="request.url"/>
> </route>
> </routes>
>
> And in rewrite.xml
>
> <!--Only supporting Solr queries via Knox -->
> <rule dir="IN" name="SOLR/solr/inbound/query"
> pattern="*://*:*/**/solr/{collection}/{query}?{**}">
> <rewrite template="{$serviceUrl[SOLR]}/
> {collection}/{query}?{**}"/>
> </rule>
>
> Let me know if you run into issues, I can take a look at the code.
>
> Best,
> Sandeep
>
>
> On Fri, Nov 18, 2016 at 11:05 AM, McParland, John <john.mcparl...@cgi.com>
> wrote:
>
> > Hi Sandeep,
> >
> > thanks for your help. A part of the difficulty I'm having is that the
> > "KnoxSolrIntegration" part of the URL is variable - the user sending a
> > request to Knox needs to specify it (it happens to the be the collection
> in
> > Solr which they wish to query).
> >
> > E.g. in a normal Solr Query;
> >
> > curl -X GET 'http://hdp24sandbox:8983/solr/KnoxIntegrationConfig/
> > select?q=*.*&wt=json&indent=true
> >
> > All of the pieces after the "/solr/" are variable - the user needs to
> > specify it regardless of whether going through Knox or directly to Solr.
> >
> > So my belief is a Knox equivalent of the above curl command is
> >
> > curl -k -u guest:guest-password -X GET 'https://hdp24sandbox:8443/
> > gateway/sandbox/solr/KnoxIntegrationConfig/select?
> > q=*.*&wt=json&indent=true'
> >
> > Therefore, I thought the pattern to match (in rewrite.xml) would be
> >
> > *://*.*//**/solr/**/**?{**}
> >
> > As in my head
> >
> > - first * = protocol (http/https)
> > - second * = domain name / IP address of Knox server
> > - third * = port of Knox on it's server
> >
> > - first ** = knox context and topology (gateway/sandbox in my example)
> > - second ** = Solr collection to search (KnoxIntegrationConfig in my
> > example, but something I expect the caller to provide)
> > - third ** = query type for Solr (e.g. "select")
> > - {**} = the search parameters for Solr (q=*.*&wt=json&indent=true in my
> > example)
> >
> > I suspect either (or both)
> > a) I've not captured my pattern correctly (but I think so....)
> > b) I haven't linked the service to the re-write rule to the routes in the
> > service definition correctly [as I'm not sure if/how this works from the
> > tutorial]
> >
> > Thanks,
> >
> > John McParland MIET CEng | System Architect, ODSC
> > Health Local and Scotland | CGI
> > CGI Ltd (UK)
> > Second Floor, Inovo Building, 121 George St, Glasgow, UK, G1 1RD
> > M: +44 7920 183 019
> > john.mcparl...@cgi.com | www.cgi-group.co.uk
> >
> > CGI IT UK Limited. A CGI Group Inc. Company
> > Registered Office: 250 Brook Drive, Green Park, Reading RG2 6UA,
> > United Kingdom. Registered in England & Wales - Number 947968
> >
> > CONFIDENTIALITY NOTICE: Proprietary/Confidential Information belonging to
> > CGI Group Inc. and its affiliates may be contained in this message. If
> you
> > are not a recipient indicated or intended in this message (or responsible
> > for delivery of this message to such person), or you think for any reason
> > that this message may have been addressed to you in error, you may not
> use
> > or copy or deliver this message to anyone else. In such case, you should
> > destroy this message and are asked to notify the sender by reply e-mail.
> >
> > ________________________________________
> > From: Sandeep More [moresand...@gmail.com]
> > Sent: 18 November 2016 14:11
> > To: dev@knox.apache.org
> > Subject: Re: Re-write rules not being applied to new service
> >
> > Hello John,
> >
> > I think you are missing the definitions for the path
> > '/solr/KnoxSolrIntegration'
> >
> > Looking at your definition files you could perhaps have something like
> > this:
> >
> > service.xml
> >
> > <routes>
> > <route path="/solr/**/**?{**}" >
> > <rewrite apply="SOLR/solr/inbound/query"
> > to="request.url"/>
> > </route>
> >
> > <route path="/solr/KnoxSolrIntegration**" >
> > <rewrite apply="SOLR/solr/inbound/KnoxSolrIntegration"
> > to="request.url"/>
> > </route>
> > </routes>
> >
> > And in rewrite.xml
> >
> > <!--Only supporting Solr queries via Knox -->
> > <rule dir="IN" name="SOLR/solr/inbound/query"
> > pattern="*://*:*/**/solr/{collection}/{query}?{**}">
> > <rewrite template="{$serviceUrl[SOLR]}/
> > {collection}/{query}?{**}"/>
> > </rule>
> > <rule dir="IN" name="SOLR/solr/inbound/KnoxSolrIntegration"
> > pattern="*://*:*/**/solr/KnoxSolrIntegration**">
> > <rewrite template="{$serviceUrl[SOLR]}/KnoxSolrIntegration{**}"/>
> > </rule>
> >
> > Looks like you are missing the <route> for 'SOLR/solr/inbound/query' in
> > your service.xml file (like in the above example).
> >
> > Couple of things that helped me were
> > 1) Try to keep the rules as tight as possible e.g.
> > /solr/KnoxSolrIntegration**
> > instead of /solr/** (provided an example with rule #2 in the above
> example)
> > 2) In your Service.xml file you have multiple <routes>, you could
> > consolidate them into single <routes>
> >
> > I haven't tested locally it so apologies in advance if it does not work.
> >
> >
> > Hope that helps !
> >
> > Best,
> > Sandeep
> >
> > On Fri, Nov 18, 2016 at 7:13 AM, McParland, John <john.mcparl...@cgi.com
> >
> > wrote:
> >
> > > Hi all,
> > >
> > > I'm working to add support for Solr's API to Apache Knox. To do so,
> I've
> > > created a service.xml and rewrite.xml, and deployed to
> > > ${GATEWAY_HOME}/data/conf/services/solr/5.5.0
> > >
> > > See the files here: https://github.com/mcparlandjc
> > > gi/knox/tree/KNOX-528/gateway-service-definitions/src/main/r
> > > esources/services/solr/5.5.0
> > > I also wrote a test to check that the re-write rules work as expected
> > (see
> > > https://github.com/mcparlandjcgi/knox/blob/KNOX-528/gateway-
> > > provider-rewrite/src/test/java/org/apache/hadoop/
> > > gateway/filter/rewrite/api/UrlRewriteProcessorTest.java#L281)
> > >
> > > I have defined the Solr service in my sandbox topology like so;
> > >
> > > <service>
> > > <role>SOLR</role>
> > > <url>http://sandbox.hortonworks.com:8983/solr</url>
> > > </service>
> > >
> > > However when I attempt to execute a Solr Query via Knox, I find that
> the
> > > rule isn't applied and indeed, no matching rule is found at all.
> > >
> > > 2016-11-18 12:04:23,221 TRACE gateway.access
> (AccessHandler.java:log(49))
> > > - |||158.234.220.6|GET|/gateway/sandbox/solr/KnoxSolrIntegrati
> > > on/select?q=*.*&wt=json&indent=true|-1|200|0|6
> > > 2016-11-18 12:04:23,231 TRACE http.request (TraceRequest.java:
> > traceRequestDetails(66))
> > > - ||82e1c7f7-4eb5-499d-8f29-7602652079f1|Request=GET
> > > /gateway/sandbox/solr/KnoxSolrIntegration/select?q=*
> > .*&wt=json&indent=true
> > > Header[User-Agent]=curl/7.47.0
> > > Header[Authorization]=Basic Z3Vlc3Q6Z3Vlc3QtcGFzc3dvcmQ=
> > > Header[Accept]=*/*
> > > Header[Host]=hdp24sandbox:8443
> > > 2016-11-18 12:04:23,786 DEBUG hadoop.gateway
> > (GatewayFilter.java:doFilter(116))
> > > - Received request: GET /solr/KnoxSolrIntegration/select
> > > 2016-11-18 12:04:23,888 TRACE hadoop.gateway (UrlRewriteProcessor.java:
> > rewrite(177))
> > > - No rule matching URL: hdp24sandbox:8443, direction: IN
> > > 2016-11-18 12:04:23,889 TRACE hadoop.gateway (UrlRewriteProcessor.java:
> > rewrite(177))
> > > - No rule matching URL: 8443, direction: IN
> > > 2016-11-18 12:04:23,889 TRACE hadoop.gateway (UrlRewriteProcessor.java:
> > rewrite(177))
> > > - No rule matching URL: https, direction: IN
> > > 2016-11-18 12:04:25,052 TRACE hadoop.gateway (UrlRewriteProcessor.java:
> > rewrite(177))
> > > - No rule matching URL: https://hdp24sandbox:8443/
> gateway/sandbox/solr/
> > KnoxSolrIntegrati
> > > on/select?q=*.*&wt=json&indent=true, direction: IN
> > > 2016-11-18 12:04:25,062 TRACE hadoop.gateway (UrlRewriteProcessor.java:
> > rewrite(177))
> > > - No rule matching URL: 158.234.220.6, direction: IN
> > > 2016-11-18 12:04:25,062 TRACE hadoop.gateway (UrlRewriteProcessor.java:
> > rewrite(177))
> > > - No rule matching URL: https, direction: IN
> > > 2016-11-18 12:04:25,063 TRACE hadoop.gateway (UrlRewriteProcessor.java:
> > rewrite(177))
> > > - No rule matching URL: 8443, direction: IN
> > > 2016-11-18 12:04:25,063 TRACE hadoop.gateway (UrlRewriteProcessor.java:
> > rewrite(177))
> > > - No rule matching URL: hdp24sandbox:8443, direction: IN
> > > 2016-11-18 12:04:25,063 TRACE hadoop.gateway (UrlRewriteProcessor.java:
> > rewrite(177))
> > > - No rule matching URL: hdp24sandbox, direction: IN
> > > 2016-11-18 12:04:25,063 TRACE hadoop.gateway (UrlRewriteProcessor.java:
> > rewrite(177))
> > > - No rule matching URL: /gateway/sandbox, direction: IN
> > > 2016-11-18 12:04:25,064 TRACE hadoop.gateway (UrlRewriteProcessor.java:
> > rewrite(177))
> > > - No rule matching URL: curl/7.47.0, direction: IN
> > > 2016-11-18 12:04:25,069 TRACE hadoop.gateway (UrlRewriteProcessor.java:
> > rewrite(177))
> > > - No rule matching URL: Basic Z3Vlc3Q6Z3Vlc3QtcGFzc3dvcmQ=, direction:
> IN
> > > 2016-11-18 12:04:25,070 TRACE hadoop.gateway (UrlRewriteProcessor.java:
> > rewrite(177))
> > > - No rule matching URL: */*, direction: IN
> > > 2016-11-18 12:04:25,070 TRACE hadoop.gateway (UrlRewriteProcessor.java:
> > rewrite(177))
> > > - No rule matching URL: hdp24sandbox:8443, direction: IN
> > > 2016-11-18 12:04:25,072 DEBUG hadoop.gateway (DefaultDispatch.java:
> > executeOutboundRequest(120))
> > > - Dispatch request: GET https://hdp24sandbox:8443/gate
> > > way/sandbox/solr/KnoxSolrIntegration/select?q=*.*&wt=json&indent=true
> > >
> > > What I expect at the end is the Dispatch request to go to
> > >
> > > http://sandbox.hortonworks.com:8983/solr/KnoxSolrIntegration
> > > /select?q=*.*&wt=json&indent=true
> > >
> > > Is there a missing relationship or problem in my service/rules which
> > > prevents the rule being matched with the request URL?
> > >
> > > Thanks,
> > >
> > > John McParland MIET CEng | System Architect, ODSC
> > > Health Local and Scotland | CGI
> > > CGI Ltd (UK)
> > > Second Floor, Inovo Building, 121 George St, Glasgow, UK, G1 1RD
> > > M: +44 7920 183 019
> > > john.mcparl...@cgi.com | www.cgi-group.co.uk
> > >
> > > CGI IT UK Limited. A CGI Group Inc. Company
> > > Registered Office: 250 Brook Drive, Green Park, Reading RG2 6UA,
> > > United Kingdom. Registered in England & Wales - Number 947968
> > >
> > > CONFIDENTIALITY NOTICE: Proprietary/Confidential Information belonging
> to
> > > CGI Group Inc. and its affiliates may be contained in this message. If
> > you
> > > are not a recipient indicated or intended in this message (or
> responsible
> > > for delivery of this message to such person), or you think for any
> reason
> > > that this message may have been addressed to you in error, you may not
> > use
> > > or copy or deliver this message to anyone else. In such case, you
> should
> > > destroy this message and are asked to notify the sender by reply
> e-mail.
> > >
> >
>
