Hi All, I tried to change the content-type of the http header in the ElasticsearchDispatcher java class, but doesn't work :(
Anyone can give help where we should change the content-type for make "application/x-ndjson" in class of ElasticsearchDispatcher (attached) Regards Rabii Le mar. 25 sept. 2018 à 16:20, rabii lamriq <[email protected]> a écrit : > I am not understand how to apply this for content-type. > > Le mar. 25 sept. 2018 à 15:18, Sandeep Moré <[email protected]> a > écrit : > >> This is an example in Webhdfs service def. >> https://github.com/apache/knox/blob/6e7266ad3649e1804ee869afed04f79a71e48b7d/gateway-service-definitions/src/main/resources/services/webhdfs/2.4.0/rewrite.xml#L64 >> >> >> >> On Tue, Sep 25, 2018 at 5:37 AM rabii lamriq <[email protected]> wrote: >> >>> Hi Sandeep, >>> >>> Yes, I know that is possible, but I don't know how to rewrite the rule >>> to access to the header and change the Content-type, I can't find any thing >>> related to this customization. >>> >>> Regards >>> Rabii >>> >>> Le lun. 24 sept. 2018 à 17:19, Sandeep Moré <[email protected]> a >>> écrit : >>> >>>> Yes, you can use the rewrite rules to update Headers, I don't remember >>>> exactly which service uses it but there should be few services that rewrite >>>> headers. >>>> >>>> Best, >>>> Sandeep >>>> >>>> On Mon, Sep 24, 2018 at 10:45 AM rabii lamriq <[email protected]> wrote: >>>> >>>>> >>>>> Rewrite rules for knox (link >>>>> <https://cwiki.apache.org/confluence/display/KNOX/2017/08/14/Understanding+Rewrite+Rules+for+Apache+Knox#UnderstandingRewriteRulesforApacheKnox-JSONURLRewriteFIlter>) >>>>> can help me to change header? is it possible? >>>>> >>>>> Le lun. 24 sept. 2018 à 16:43, rabii lamriq <[email protected]> a >>>>> écrit : >>>>> >>>>>> Hello >>>>>> >>>>>> Any help please?? >>>>>> >>>>>> Le lun. 24 sept. 2018 à 09:31, rabii lamriq <[email protected]> a >>>>>> écrit : >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> thanks every One, for your feedbacks, >>>>>>> >>>>>>> Yes, ES accept both application/json and application/x-ndjson, there >>>>>>> is no problem in ES ni in Grafana, when i test my query with curl to >>>>>>> send >>>>>>> POST to ES, it work well ether json or x-ndjson. >>>>>>> >>>>>>> However, through Knox Gateway, it work only for x-ndjson, I don't >>>>>>> know why. so, since Grafana send their POST with only json and not >>>>>>> x-ndjson, the scenario of Grafana -> Knox -> ES with multi search >>>>>>> doesn't >>>>>>> work. >>>>>>> >>>>>>> I think we have two solution: >>>>>>> >>>>>>> 1- make some thing to force Grafana change header of POST and put >>>>>>> x-ndjson instead of json, or >>>>>>> 2- rewrite Header by Knox to forwarded with x-ndjson to ES. >>>>>>> >>>>>>> If some one have any idea of how to make this or other solution. >>>>>>> >>>>>>> >>>>>>> Regards >>>>>>> Rabii >>>>>>> >>>>>>> Le ven. 21 sept. 2018 à 19:21, Kevin Risden <[email protected]> a >>>>>>> écrit : >>>>>>> >>>>>>>> https://github.com/elastic/elasticsearch/issues/25718 >>>>>>>> >>>>>>>> Elasticsearch endpoints should take both application/json and >>>>>>>> application/x-ndjson. There might be something else going on here. >>>>>>>> That is >>>>>>>> why I asked for the Elasticsearch logs. >>>>>>>> >>>>>>>> It might not be necessary to rewrite the content type. >>>>>>>> >>>>>>>> Kevin Risden >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Sep 21, 2018 at 1:17 PM rabii lamriq <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> My problem is how to change the content-type=application/json to >>>>>>>>> application/x-ndjson by rewriting the header in the knox. >>>>>>>>> >>>>>>>>> Because grafana send their query with content-type=application/json >>>>>>>>> and knox return status 400, >>>>>>>>> >>>>>>>>> Hower when i reproduce the problem by curl comande, I noticed that >>>>>>>>> if i replace json by x-ndjson through knox it wirk well. >>>>>>>>> >>>>>>>>> Sence i can't change the content-type in grafana, is this possible >>>>>>>>> by rewriting the header in knox to replace json by x-ndjson. >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> Rabii >>>>>>>>> >>>>>>>>> Le ven. 21 sept. 2018 à 18:05, >>>>>>>>> Kevin Risden >>>>>>>>> <[email protected]> a écrit : >>>>>>>>> >>>>>>>>>> Do you have the error from Elasticsearch logs? It might explain >>>>>>>>>> if Knox or Grafana is sending a bad request. >>>>>>>>>> >>>>>>>>>> Kevin Risden >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Sep 21, 2018 at 12:03 PM Sandeep Moré < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> I don't see any issues from Knox side, what error do you see in >>>>>>>>>>> Grafana ? I think this could be because Grafana is expecting >>>>>>>>>>> certain >>>>>>>>>>> content-type. >>>>>>>>>>> >>>>>>>>>>> On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Sandeep >>>>>>>>>>>> >>>>>>>>>>>> thanks for your feedback, below is the deboug log for this curl >>>>>>>>>>>> >>>>>>>>>>>> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway >>>>>>>>>>>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: >>>>>>>>>>>> https://<host-knox>:8443/gateway/default/es5/<index_name>/_msearch, >>>>>>>>>>>> direction: IN via explicit rule: ES5/inbound/query to URL: http:// >>>>>>>>>>>> <host-elast>:9200/<index_name>/_msearch >>>>>>>>>>>> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway >>>>>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch >>>>>>>>>>>> request: >>>>>>>>>>>> POST http:// >>>>>>>>>>>> <host-elast>:9200/<index_name>/_msearch?doAs=<u_user> >>>>>>>>>>>> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway >>>>>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch >>>>>>>>>>>> response >>>>>>>>>>>> status: 400 >>>>>>>>>>>> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway >>>>>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using >>>>>>>>>>>> explicit >>>>>>>>>>>> character set UTF-8 for entity of type application/json >>>>>>>>>>>> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway >>>>>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound >>>>>>>>>>>> response entity content type: application/json; charset=UTF-8 >>>>>>>>>>>> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway >>>>>>>>>>>> (GatewayFilter.java:doFilter(116)) - Received request: POST >>>>>>>>>>>> /es5/<index_name>/_msearch >>>>>>>>>>>> 2018-09-18 17:23:54,521 INFO hadoop.gateway >>>>>>>>>>>> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true >>>>>>>>>>>> >>>>>>>>>>>> Regards >>>>>>>>>>>> Rabii >>>>>>>>>>>> >>>>>>>>>>>> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré < >>>>>>>>>>>> [email protected]> a écrit : >>>>>>>>>>>> >>>>>>>>>>>>> Hello Rab, >>>>>>>>>>>>> >>>>>>>>>>>>> Interesting, do you have the debug logs for this ? they will >>>>>>>>>>>>> tell us exactly what is happening. >>>>>>>>>>>>> >>>>>>>>>>>>> Best, >>>>>>>>>>>>> Sandeep >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>> >>>>>>>>>>>>>> From Grafana, I Post multisearch query to Elasticsearch using >>>>>>>>>>>>>> the bulk API with the newline delimited JSON (NDJSON) format. >>>>>>>>>>>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html >>>>>>>>>>>>>> like this: >>>>>>>>>>>>>> >>>>>>>>>>>>>> curl -u myuser -i -H “Content-Type: application/json” -XPOST " >>>>>>>>>>>>>> http://host_ES:900/index/_msearch"; --data-binary ’ >>>>>>>>>>>>>> { first json doc} >>>>>>>>>>>>>> {second json doc} >>>>>>>>>>>>>> ’ >>>>>>>>>>>>>> when bypassing Knox, The query work well either we made json >>>>>>>>>>>>>> or x-ndjson, >>>>>>>>>>>>>> However, through Knox gateway, if the >>>>>>>>>>>>>> content-type=application/json, the query retun this error of 400: >>>>>>>>>>>>>> >>>>>>>>>>>>>> HTTP/1.1 400 Bad Request >>>>>>>>>>>>>> content-type: application/json >>>>>>>>>>>>>> >>>>>>>>>>>>>> But when we replace json by x-ndjson like mentioned in the >>>>>>>>>>>>>> bulk API, it work well. >>>>>>>>>>>>>> >>>>>>>>>>>>>> My problem is that Grafana send their query with the >>>>>>>>>>>>>> content-type=application/json and not x-ndjson, for this, it is >>>>>>>>>>>>>> possible to >>>>>>>>>>>>>> make any thing on Knox to solve this problem? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards >>>>>>>>>>>>>> Rab >>>>>>>>>>>>>> >>>>>>>>>>>>>
ElasticsearchDispatch.java
Description: Binary data
