The existing processors will not be refactored to use it. I'm writing new ones.
On Tue, Apr 10, 2018 at 12:08 AM, Jon Logan <[email protected]> wrote: > Thanks, a few notes -- as far as not using AWS ES, we have been having > issues with standard-ES, mainly the lack of encryption and authentication > without the X-Pack (yes, I could proxy but that seems like a band-aid).. > note that AWS VPCs are not encrypted internally, and any machine that you > want to have updates must have a public-facing IP address (no VPC-only). > > This could be done using the AWS ES SDK, but it seems like that's more > trouble than it's worth...the AWS ES endpoints are identical, yet support > the additional signing attribute. It would seem unnecessary to have two > disjoint implementations, rather than just adding the signatures to the > request...similar to this (I found this on Google, so no guarantees!) > https://gist.github.com/missingfaktor/664b01b7989e91033657e2e703f14ed6 . I > think the only piece of the AWS SDK this would require would be the things > to get the credentials, and the signing part itself. > > The new ES client service looks much easier to use than the previous one. > Is this going to eventually be used by the existing processors (ex. > FetchEsHttp, PutEsHttp, etc)? I suppose this could be even have a > TransportClient implementation and merge the FetchEs FetchEsHttp and > FetchEs5 (and put, etc) processors? > > On Mon, Apr 9, 2018 at 9:47 PM, Mike Thomsen <[email protected]> > wrote: > > > Related, interesting take on AWS ES: > > > > https://code972.com/blog/2017/12/111-why-you-shouldnt-use- > > aws-elasticsearch-service > > > > I saw some other critical posts that went into detail on other issues. > > Considering how easy it is to stand up ES yourself, I think it would be > > worthwhile to strongly consider going that route. > > > > On Mon, Apr 9, 2018 at 9:02 PM, Otto Fowler <[email protected]> > > wrote: > > > > > Sorry, just like the gateway api class, it is for managing ES not > > calling. > > > > > > > > > On April 9, 2018 at 20:29:33, Otto Fowler ([email protected]) > > wrote: > > > > > > The aws java sdk has a purpose built ElasticSearchClient class. > > > The way to do this and be consistent would be to add a new nifi-aws > > > processor for the ES client, > > > as there is for s3 and dynamoDB etc. > > > > > > https://github.com/apache/nifi/pull/2588 is my outstanding PR for > > > HttpInvoke for AWS Gateway Web APIs. > > > A similar approach being having version of the ElasticSearchHttp but in > > the > > > aws nar. > > > > > > > > > > > > On April 9, 2018 at 19:11:45, Jon Logan ([email protected]) wrote: > > > > > > Hi All, > > > > > > We are looking to use the built-in Nifi Elasticsearch Http processors > > with > > > signed AWS ES requests. I tried to extend them to do so, but ran into > > > issues with things being in-extensible (private-static-final in some > > > cases), and was wondering if this would be something that would be > > > considered to be merged into the baseline? > > > > > > There's two main ways I saw doing it -- either modifying the existing > ES > > > code to allow for it, or making new AWS-specific extended adapters to > do > > > it. In the former, it might require dependencies between the ES code > and > > > the AWS code for optional CredentialProviders, etc., and am not sure > how > > > isolated you all try to keep things. > > > > > > In either case, AWS essentially adds an HTTP-header signature to > > > authenticate the request against your ES instance. The easiest path to > do > > > this seems to be writing a bridge between the ES OkHttpClient requests > > and > > > the AWS requests to generate the signature correctly. > > > > > > > > > Just wanted to get some feedback to be sure this wouldn't be a waste of > > > time. > > > > > > > > > Thanks! > > > Jon > > > > > >
