Agreed Matt. I do think Jira is probably the most sensible place at this
point. I typically link my PRs back to the Jira in the main comments to
make access easy and fast anyhow.

On Thu, Oct 12, 2017 at 10:50 AM, Matt Foley <mfo...@hortonworks.com> wrote:

> Github also allows file attachments, via comments on PRs.  Not necessarily
> intuitive for a “Discussion”.  There would be more sensible places to put
> files for discussion in github, such as “wiki” or “issues”, but those
> aren’t enabled on apache projects in github.
>
> On 10/11/17, 10:39 PM, "Michael Miklavcic" <michael.miklav...@gmail.com>
> wrote:
>
>     We've generally preferred communication workflows via Github and the
>     mailing list rather than Jira for most things on this project, but
> you're
>     right that we could probably leverage it for sharing attachments to
> the dev
>     list.
>
>     On Wed, Oct 11, 2017 at 9:54 PM, Matt Foley <mfo...@hortonworks.com>
> wrote:
>
>     > You can avoid the permission issues by attaching it to an Apache
> jira.
>     >
>     > On 10/11/17, 6:10 PM, "James Sirota" <jsir...@apache.org> wrote:
>     >
>     >     I can't see it.  You probably want to link to a google drive
>     >
>     >     11.10.2017, 18:01, "Michael Miklavcic" <
> michael.miklav...@gmail.com>:
>     >     > I attached a PDF - shows up on my end. Is that not coming
> through?
>     >     >
>     >     > On Wed, Oct 11, 2017 at 6:42 PM, Otto Fowler <
>     > ottobackwa...@gmail.com>
>     >     > wrote:
>     >     >
>     >     >>  I think there is a missing attachment?
>     >     >>
>     >     >>  On October 11, 2017 at 20:22:33, Michael Miklavcic (
>     >     >>  michael.miklav...@gmail.com) wrote:
>     >     >>
>     >     >>  For community reference, here is a class diagram that
> depicts our
>     > current
>     >     >>  Metron 0.4.1 dependencies, for both prod and test code,
> against
>     > the old ES
>     >     >>  client APIs along with an "after" diagram showing the world
> with
>     > the new
>     >     >>  client. Feedback welcome.
>     >     >>
>     >     >>  On Fri, Oct 6, 2017 at 8:13 AM, Casey Stella <
> ceste...@gmail.com>
>     > wrote:
>     >     >>
>     >     >>>  Yeah, I agree with what Michael "fine whine" Miklavcic
> said; I'm
>     > in favor
>     >     >>>  of the high level client.
>     >     >>>
>     >     >>>  On Thu, Oct 5, 2017 at 3:35 PM, Michael Miklavcic <
>     >     >>>  michael.miklav...@gmail.com> wrote:
>     >     >>>
>     >     >>>  > Justin, thanks for the feedback! I'm inclined to agree
> with you
>     > about
>     >     >>>  using
>     >     >>>  > the high level client. It's a bummer that we still need
> to do
>     > jar
>     >     >>>  shading,
>     >     >>>  > but I think that's a reasonable short term sacrifice
>     > considering the
>     >     >>>  other
>     >     >>>  > benefits. And they're angling towards slowly removing the
> ES
>     > core dep
>     >     >>>  over
>     >     >>>  > time anyhow so, like myself, this will get better with
> age.
>     >     >>>  >
>     >     >>>  > On Thu, Oct 5, 2017 at 12:40 PM, Justin Leet <
>     > justinjl...@gmail.com>
>     >     >>>  > wrote:
>     >     >>>  >
>     >     >>>  > > Do we intend on (or have interest in) supporting ES
> across
>     > major
>     >     >>>  version
>     >     >>>  > > for a given version of Metron? I'm not convinced it's
> worth
>     > the work
>     >     >>>  of
>     >     >>>  > > using the low level client.
>     >     >>>  > >
>     >     >>>  > > This really only seems useful for ES clusters that are
> being
>     > used
>     >     >>>  outside
>     >     >>>  > > Metron and need to be on a different ES major version.
> Is
>     > that a use
>     >     >>>  case
>     >     >>>  > > we want/need to support? I'm willing to bet it's
>     > significantly more
>     >     >>>  work
>     >     >>>  > > and means we're modifying queries and even
> templates/mappings
>     > based on
>     >     >>>  > what
>     >     >>>  > > ES version we're interacting with (e.g. meta alerts in
> 5.x can
>     >     >>>  exploit a
>     >     >>>  > > query param to not screw around with the mapping, but
> that
>     > param
>     >     >>>  doesn't
>     >     >>>  > > exist in 2.x). At that point, we're either back to
> writing
>     > for ES 2.x
>     >     >>>  or
>     >     >>>  > > writing for every version of ES.
>     >     >>>  > >
>     >     >>>  > > Unless that's something we have a demand for (or
> someone else
>     >     >>>  persuades
>     >     >>>  > me
>     >     >>>  > > otherwise), I'm in favor of using the high level
> client. It
>     > seems
>     >     >>>  like
>     >     >>>  > > it'd be easier to migrate to also, given the
> similarities
>     > API-wise to
>     >     >>>  the
>     >     >>>  > > current client we're using.
>     >     >>>  > >
>     >     >>>  > > On Thu, Oct 5, 2017 at 1:52 PM, Michael Miklavcic <
>     >     >>>  > > michael.miklav...@gmail.com> wrote:
>     >     >>>  > >
>     >     >>>  > > > I think it might help the discussion to share my
>     > impressions of
>     >     >>>  looking
>     >     >>>  > > > over the new API recommendations from ES. I've
> summarized
>     > some info
>     >     >>>  > > > provided by ES back in December 2016 regarding the
> reasons
>     > for
>     >     >>>  > switching
>     >     >>>  > > to
>     >     >>>  > > > a new client model. [1]
>     >     >>>  > > >
>     >     >>>  > > > *Summary points:*
>     >     >>>  > > >
>     >     >>>  > > > Pre-5.x had Java API - binary exchange format used for
>     > node-to-node
>     >     >>>  > > > communications.
>     >     >>>  > > > In 5.x a low level REST API was added. Now there's
> also a
>     > high level
>     >     >>>  > REST
>     >     >>>  > > > client that handles request marshalling and response
>     > un-marshalling.
>     >     >>>  > > >
>     >     >>>  > > > *Benefits of existing Java API*
>     >     >>>  > > >
>     >     >>>  > > > 1. Theoretically faster - binary format, no JSON
> parsing
>     >     >>>  > > > 2. Hardened, used for internal ES node to node
>     > communications
>     >     >>>  > > >
>     >     >>>  > > > *Cons of Java API*
>     >     >>>  > > >
>     >     >>>  > > > 1. Benchmarks show it's not really that much faster.
>     >     >>>  > > > 2. Backwards compatibility - Java API changes often.
>     >     >>>  > > > 3. Upgrades more challenging - need to refactor
> client code
>     > for
>     >     >>>  new
>     >     >>>  > > and
>     >     >>>  > > > deprecated features.
>     >     >>>  > > > 4. Minor releases may contain breaking changes in the
> Java
>     > API
>     >     >>>  > > > 5. Client and server *should* be on same JVM version
> (not as
>     >     >>>  > important
>     >     >>>  > > > post 2.x, but still potentially necessary bc of
>     > serialization
>     >     >>>  > w/binary
>     >     >>>  > > > format)
>     >     >>>  > > > 6. Requires dependency on the entire elasticsearch
> server in
>     >     >>>  order
>     >     >>>  > to
>     >     >>>  > > > use the client. We end up shading jars.
>     >     >>>  > > >
>     >     >>>  > > > *Benefits of new REST API*
>     >     >>>  > > >
>     >     >>>  > > > 1. Upgrades
>     >     >>>  > > > 1. Breaking changes only made in major releases - "We
> are
>     > very
>     >     >>>  > > > careful with backwards compatibility on the REST
> layer where
>     >     >>>  > > breaking
>     >     >>>  > > > changes are made only in major releases."
>     >     >>>  > > > 2. "The REST interface is much more stable and can be
>     > upgraded
>     >     >>>  > out
>     >     >>>  > > of
>     >     >>>  > > > step with the Elasticsearch cluster."
>     >     >>>  > > > 2. REST client and server can be on different JVM's
>     >     >>>  > > > 3. Dependencies for the low level client are very
> slim. No
>     > need
>     >     >>>  for
>     >     >>>  > > > shading.
>     >     >>>  > > > 4. The RestHighLevelClient supports the same request
> and
>     > response
>     >     >>>  > > > objects as the TransportClient
>     >     >>>  > > > 5. Can be secured via HTTPS
>     >     >>>  > > >
>     >     >>>  > > > There are some additional benefits to the new API,
> however
>     > they
>     >     >>>  depend
>     >     >>>  > on
>     >     >>>  > > > whether we choose to go with the high or low level
> client.
>     > More
>     >     >>>  > comments
>     >     >>>  > > > below.
>     >     >>>  > > >
>     >     >>>  > > > *Cons of new API*
>     >     >>>  > > >
>     >     >>>  > > > 1. Dependencies - The high level client still
> requires the
>     > full
>     >     >>>  ES
>     >     >>>  > > > dependency, though this will slim down in future
> releases.
>     >     >>>  > > >
>     >     >>>  > > > *Other comments specific to Metron*
>     >     >>>  > > >
>     >     >>>  > > > There's a question of whether we should use the low
> or high
>     > level
>     >     >>>  REST
>     >     >>>  > > > client. The main differences between the two are how
> they
>     > handle lib
>     >     >>>  > > > dependencies and marshaling/unmarshaling. The low
> level
>     > client
>     >     >>>  cleans
>     >     >>>  > up
>     >     >>>  > > > the dependencies dramatically, whereas the high level
>     > client still
>     >     >>>  > > requires
>     >     >>>  > > > you to depend on elasticsearch core. On the other
> hand, the
>     > low
>     >     >>>  level
>     >     >>>  > > > client does no work to handle marshaling/unmarshaling
> the
>     >     >>>  > > > requests/responses from the HTTP calls while the high
> level
>     > client
>     >     >>>  > > handles
>     >     >>>  > > > this for you and exposes api-specific methods. The
> high
>     > level client
>     >     >>>  > > > accepts the same request arguments as the
> TransportClient
>     > and
>     >     >>>  returns
>     >     >>>  > the
>     >     >>>  > > > same response objects. One more thing to note is that
> the
>     > low level
>     >     >>>  > > client
>     >     >>>  > > > claims to be compatible with all versions of ES
> whereas the
>     > high
>     >     >>>  level
>     >     >>>  > > > client appears to be only major version compatible.
>     >     >>>  > > >
>     >     >>>  > > > "The 5.6 client can communicate with any 5.6.x
>     > Elasticsearch node.
>     >     >>>  > > Previous
>     >     >>>  > > > 5.x minor versions like 5.5.x, 5.4.x etc. are not
> (fully)
>     >     >>>  supported."
>     >     >>>  > [2]
>     >     >>>  > > >
>     >     >>>  > > > Just as an example, here's a simple comparison of an
> index
>     > request
>     >     >>>  in
>     >     >>>  > the
>     >     >>>  > > > low and high level API's.
>     >     >>>  > > >
>     >     >>>  > > > *Low Level*
>     >     >>>  > > >
>     >     >>>  > > > Map<String, String> params = Collections.emptyMap();
>     >     >>>  > > > String jsonString = "{" +
>     >     >>>  > > > "\"user\":\"kimchy\"," +
>     >     >>>  > > > "\"postDate\":\"2013-01-30\"," +
>     >     >>>  > > > "\"message\":\"trying out Elasticsearch\"" +
>     >     >>>  > > > "}";
>     >     >>>  > > > HttpEntity entity = new NStringEntity(jsonString,
>     >     >>>  > > > ContentType.APPLICATION_JSON);
>     >     >>>  > > > Response response = restClient.performRequest("PUT",
>     >     >>>  "/posts/doc/1",
>     >     >>>  > > > params, entity);
>     >     >>>  > > >
>     >     >>>  > > > *High Level*
>     >     >>>  > > >
>     >     >>>  > > > IndexRequest indexRequest = new IndexRequest("posts",
>     > "doc", "1")
>     >     >>>  > > > .source("user", "kimchy",
>     >     >>>  > > > "postDate", new Date(),
>     >     >>>  > > > "message", "trying out Elasticsearch");
>     >     >>>  > > >
>     >     >>>  > > > *Note*: there are a few ways to do this with the high
> level
>     > API, but
>     >     >>>  > this
>     >     >>>  > > > was the most concise for me to offer a comparison of
>     > benefits over
>     >     >>>  the
>     >     >>>  > > low
>     >     >>>  > > > level API.
>     >     >>>  > > >
>     >     >>>  > > > *Thoughts/Recommendations*: I do think we should
> migrate to
>     > the new
>     >     >>>  > API.
>     >     >>>  > > I
>     >     >>>  > > > think the question is which of the new APIs we should
> use.
>     > The high
>     >     >>>  > level
>     >     >>>  > > > client seems to shield us from having to deal with
>     > constructing
>     >     >>>  special
>     >     >>>  > > > JSON handling code, whereas the low level client
> handles all
>     >     >>>  versions
>     >     >>>  > of
>     >     >>>  > > > ES. I don't have a good feel (yet) for just how much
> work
>     > it would
>     >     >>>  > > require
>     >     >>>  > > > to use the low level API, or how difficult it would
> be to
>     > add new
>     >     >>>  > request
>     >     >>>  > > > features in the future. Actually, we could probably
> leverage
>     >     >>>  existing
>     >     >>>  > > code
>     >     >>>  > > > we have for dealing with JSON maps, so this might be
> really
>     > easy.
>     >     >>>  > Someone
>     >     >>>  > > > with more experience in Metron's ES client use might
> have a
>     > better
>     >     >>>  idea
>     >     >>>  > > of
>     >     >>>  > > > the pros and cons to this. The high level client
> appears to
>     > handle
>     >     >>>  > > > everything all JSON manipulation for us, but we lose
> the
>     > benefit of
>     >     >>>  a
>     >     >>>  > > > simpler dependency tree and support for all versions
> of ES.
>     > My only
>     >     >>>  > > concern
>     >     >>>  > > > with "supports all versions" is that I have to imagine
>     > there are
>     >     >>>  > specific
>     >     >>>  > > > calls that we'd have to be careful of when
> constructing the
>     > JSON
>     >     >>>  > > requests,
>     >     >>>  > > > so it's unclear to me if this is better or worse in
> the end.
>     >     >>>  > > >
>     >     >>>  > > > Best,
>     >     >>>  > > > Mike
>     >     >>>  > > >
>     >     >>>  > > >
>     >     >>>  > > > 1. https://www.elastic.co/blog/state-of-the-official-
>     >     >>>  > > > elasticsearch-java-clients
>     >     >>>  > > > 2. https://www.elastic.co/guide/
>     > en/elasticsearch/client/java-
>     >     >>>  > > > rest/current/java-rest-high-compatibility.html
>     >     >>>  > > > <https://www.elastic.co/guide/
> en/elasticsearch/client/java-
>     >     >>>  > > > rest/current/java-rest-high-compatibility.html>
>     >     >>>  > > >
>     >     >>>  > > >
>     >     >>>  > > >
>     >     >>>  > > >
>     >     >>>  > > > On Wed, Sep 27, 2017 at 8:03 PM, Michael Miklavcic <
>     >     >>>  > > > michael.miklav...@gmail.com> wrote:
>     >     >>>  > > >
>     >     >>>  > > > > I am working on upgrading Elasticsearch and Kibana.
> There
>     > are
>     >     >>>  quite a
>     >     >>>  > > few
>     >     >>>  > > > > changes involved with this vix. I believe I'm mostly
>     > finished with
>     >     >>>  > the
>     >     >>>  > > > > Ambari mpack side of things, however we currently
> only
>     > support one
>     >     >>>  > > > version
>     >     >>>  > > > > with no backwards compatibility. What is the
> community's
>     > thoughts
>     >     >>>  on
>     >     >>>  > > > this?
>     >     >>>  > > > >
>     >     >>>  > > > > Here is some work contributed to the community that
> I'm
>     >     >>>  referencing
>     >     >>>  > > while
>     >     >>>  > > > > working on this upgrade -
> https://github.com/apache/
>     >     >>>  > > > metron/pull/619/files
>     >     >>>  > > > >
>     >     >>>  > > > > Best,
>     >     >>>  > > > > Michael Miklavcic
>     >     >>>  > > > >
>     >     >>>  > > >
>     >     >>>  > >
>     >     >>>  >
>     >
>     >     -------------------
>     >     Thank you,
>     >
>     >     James Sirota
>     >     PMC- Apache Metron
>     >     jsirota AT apache DOT org
>     >
>     >
>     >
>     >
>
>
>

Reply via email to