Agreed Matt. I do think Jira is probably the most sensible place at this point. I typically link my PRs back to the Jira in the main comments to make access easy and fast anyhow.
On Thu, Oct 12, 2017 at 10:50 AM, Matt Foley <mfo...@hortonworks.com> wrote: > Github also allows file attachments, via comments on PRs. Not necessarily > intuitive for a “Discussion”. There would be more sensible places to put > files for discussion in github, such as “wiki” or “issues”, but those > aren’t enabled on apache projects in github. > > On 10/11/17, 10:39 PM, "Michael Miklavcic" <michael.miklav...@gmail.com> > wrote: > > We've generally preferred communication workflows via Github and the > mailing list rather than Jira for most things on this project, but > you're > right that we could probably leverage it for sharing attachments to > the dev > list. > > On Wed, Oct 11, 2017 at 9:54 PM, Matt Foley <mfo...@hortonworks.com> > wrote: > > > You can avoid the permission issues by attaching it to an Apache > jira. > > > > On 10/11/17, 6:10 PM, "James Sirota" <jsir...@apache.org> wrote: > > > > I can't see it. You probably want to link to a google drive > > > > 11.10.2017, 18:01, "Michael Miklavcic" < > michael.miklav...@gmail.com>: > > > I attached a PDF - shows up on my end. Is that not coming > through? > > > > > > On Wed, Oct 11, 2017 at 6:42 PM, Otto Fowler < > > ottobackwa...@gmail.com> > > > wrote: > > > > > >> I think there is a missing attachment? > > >> > > >> On October 11, 2017 at 20:22:33, Michael Miklavcic ( > > >> michael.miklav...@gmail.com) wrote: > > >> > > >> For community reference, here is a class diagram that > depicts our > > current > > >> Metron 0.4.1 dependencies, for both prod and test code, > against > > the old ES > > >> client APIs along with an "after" diagram showing the world > with > > the new > > >> client. Feedback welcome. > > >> > > >> On Fri, Oct 6, 2017 at 8:13 AM, Casey Stella < > ceste...@gmail.com> > > wrote: > > >> > > >>> Yeah, I agree with what Michael "fine whine" Miklavcic > said; I'm > > in favor > > >>> of the high level client. > > >>> > > >>> On Thu, Oct 5, 2017 at 3:35 PM, Michael Miklavcic < > > >>> michael.miklav...@gmail.com> wrote: > > >>> > > >>> > Justin, thanks for the feedback! I'm inclined to agree > with you > > about > > >>> using > > >>> > the high level client. It's a bummer that we still need > to do > > jar > > >>> shading, > > >>> > but I think that's a reasonable short term sacrifice > > considering the > > >>> other > > >>> > benefits. And they're angling towards slowly removing the > ES > > core dep > > >>> over > > >>> > time anyhow so, like myself, this will get better with > age. > > >>> > > > >>> > On Thu, Oct 5, 2017 at 12:40 PM, Justin Leet < > > justinjl...@gmail.com> > > >>> > wrote: > > >>> > > > >>> > > Do we intend on (or have interest in) supporting ES > across > > major > > >>> version > > >>> > > for a given version of Metron? I'm not convinced it's > worth > > the work > > >>> of > > >>> > > using the low level client. > > >>> > > > > >>> > > This really only seems useful for ES clusters that are > being > > used > > >>> outside > > >>> > > Metron and need to be on a different ES major version. > Is > > that a use > > >>> case > > >>> > > we want/need to support? I'm willing to bet it's > > significantly more > > >>> work > > >>> > > and means we're modifying queries and even > templates/mappings > > based on > > >>> > what > > >>> > > ES version we're interacting with (e.g. meta alerts in > 5.x can > > >>> exploit a > > >>> > > query param to not screw around with the mapping, but > that > > param > > >>> doesn't > > >>> > > exist in 2.x). At that point, we're either back to > writing > > for ES 2.x > > >>> or > > >>> > > writing for every version of ES. > > >>> > > > > >>> > > Unless that's something we have a demand for (or > someone else > > >>> persuades > > >>> > me > > >>> > > otherwise), I'm in favor of using the high level > client. It > > seems > > >>> like > > >>> > > it'd be easier to migrate to also, given the > similarities > > API-wise to > > >>> the > > >>> > > current client we're using. > > >>> > > > > >>> > > On Thu, Oct 5, 2017 at 1:52 PM, Michael Miklavcic < > > >>> > > michael.miklav...@gmail.com> wrote: > > >>> > > > > >>> > > > I think it might help the discussion to share my > > impressions of > > >>> looking > > >>> > > > over the new API recommendations from ES. I've > summarized > > some info > > >>> > > > provided by ES back in December 2016 regarding the > reasons > > for > > >>> > switching > > >>> > > to > > >>> > > > a new client model. [1] > > >>> > > > > > >>> > > > *Summary points:* > > >>> > > > > > >>> > > > Pre-5.x had Java API - binary exchange format used for > > node-to-node > > >>> > > > communications. > > >>> > > > In 5.x a low level REST API was added. Now there's > also a > > high level > > >>> > REST > > >>> > > > client that handles request marshalling and response > > un-marshalling. > > >>> > > > > > >>> > > > *Benefits of existing Java API* > > >>> > > > > > >>> > > > 1. Theoretically faster - binary format, no JSON > parsing > > >>> > > > 2. Hardened, used for internal ES node to node > > communications > > >>> > > > > > >>> > > > *Cons of Java API* > > >>> > > > > > >>> > > > 1. Benchmarks show it's not really that much faster. > > >>> > > > 2. Backwards compatibility - Java API changes often. > > >>> > > > 3. Upgrades more challenging - need to refactor > client code > > for > > >>> new > > >>> > > and > > >>> > > > deprecated features. > > >>> > > > 4. Minor releases may contain breaking changes in the > Java > > API > > >>> > > > 5. Client and server *should* be on same JVM version > (not as > > >>> > important > > >>> > > > post 2.x, but still potentially necessary bc of > > serialization > > >>> > w/binary > > >>> > > > format) > > >>> > > > 6. Requires dependency on the entire elasticsearch > server in > > >>> order > > >>> > to > > >>> > > > use the client. We end up shading jars. > > >>> > > > > > >>> > > > *Benefits of new REST API* > > >>> > > > > > >>> > > > 1. Upgrades > > >>> > > > 1. Breaking changes only made in major releases - "We > are > > very > > >>> > > > careful with backwards compatibility on the REST > layer where > > >>> > > breaking > > >>> > > > changes are made only in major releases." > > >>> > > > 2. "The REST interface is much more stable and can be > > upgraded > > >>> > out > > >>> > > of > > >>> > > > step with the Elasticsearch cluster." > > >>> > > > 2. REST client and server can be on different JVM's > > >>> > > > 3. Dependencies for the low level client are very > slim. No > > need > > >>> for > > >>> > > > shading. > > >>> > > > 4. The RestHighLevelClient supports the same request > and > > response > > >>> > > > objects as the TransportClient > > >>> > > > 5. Can be secured via HTTPS > > >>> > > > > > >>> > > > There are some additional benefits to the new API, > however > > they > > >>> depend > > >>> > on > > >>> > > > whether we choose to go with the high or low level > client. > > More > > >>> > comments > > >>> > > > below. > > >>> > > > > > >>> > > > *Cons of new API* > > >>> > > > > > >>> > > > 1. Dependencies - The high level client still > requires the > > full > > >>> ES > > >>> > > > dependency, though this will slim down in future > releases. > > >>> > > > > > >>> > > > *Other comments specific to Metron* > > >>> > > > > > >>> > > > There's a question of whether we should use the low > or high > > level > > >>> REST > > >>> > > > client. The main differences between the two are how > they > > handle lib > > >>> > > > dependencies and marshaling/unmarshaling. The low > level > > client > > >>> cleans > > >>> > up > > >>> > > > the dependencies dramatically, whereas the high level > > client still > > >>> > > requires > > >>> > > > you to depend on elasticsearch core. On the other > hand, the > > low > > >>> level > > >>> > > > client does no work to handle marshaling/unmarshaling > the > > >>> > > > requests/responses from the HTTP calls while the high > level > > client > > >>> > > handles > > >>> > > > this for you and exposes api-specific methods. The > high > > level client > > >>> > > > accepts the same request arguments as the > TransportClient > > and > > >>> returns > > >>> > the > > >>> > > > same response objects. One more thing to note is that > the > > low level > > >>> > > client > > >>> > > > claims to be compatible with all versions of ES > whereas the > > high > > >>> level > > >>> > > > client appears to be only major version compatible. > > >>> > > > > > >>> > > > "The 5.6 client can communicate with any 5.6.x > > Elasticsearch node. > > >>> > > Previous > > >>> > > > 5.x minor versions like 5.5.x, 5.4.x etc. are not > (fully) > > >>> supported." > > >>> > [2] > > >>> > > > > > >>> > > > Just as an example, here's a simple comparison of an > index > > request > > >>> in > > >>> > the > > >>> > > > low and high level API's. > > >>> > > > > > >>> > > > *Low Level* > > >>> > > > > > >>> > > > Map<String, String> params = Collections.emptyMap(); > > >>> > > > String jsonString = "{" + > > >>> > > > "\"user\":\"kimchy\"," + > > >>> > > > "\"postDate\":\"2013-01-30\"," + > > >>> > > > "\"message\":\"trying out Elasticsearch\"" + > > >>> > > > "}"; > > >>> > > > HttpEntity entity = new NStringEntity(jsonString, > > >>> > > > ContentType.APPLICATION_JSON); > > >>> > > > Response response = restClient.performRequest("PUT", > > >>> "/posts/doc/1", > > >>> > > > params, entity); > > >>> > > > > > >>> > > > *High Level* > > >>> > > > > > >>> > > > IndexRequest indexRequest = new IndexRequest("posts", > > "doc", "1") > > >>> > > > .source("user", "kimchy", > > >>> > > > "postDate", new Date(), > > >>> > > > "message", "trying out Elasticsearch"); > > >>> > > > > > >>> > > > *Note*: there are a few ways to do this with the high > level > > API, but > > >>> > this > > >>> > > > was the most concise for me to offer a comparison of > > benefits over > > >>> the > > >>> > > low > > >>> > > > level API. > > >>> > > > > > >>> > > > *Thoughts/Recommendations*: I do think we should > migrate to > > the new > > >>> > API. > > >>> > > I > > >>> > > > think the question is which of the new APIs we should > use. > > The high > > >>> > level > > >>> > > > client seems to shield us from having to deal with > > constructing > > >>> special > > >>> > > > JSON handling code, whereas the low level client > handles all > > >>> versions > > >>> > of > > >>> > > > ES. I don't have a good feel (yet) for just how much > work > > it would > > >>> > > require > > >>> > > > to use the low level API, or how difficult it would > be to > > add new > > >>> > request > > >>> > > > features in the future. Actually, we could probably > leverage > > >>> existing > > >>> > > code > > >>> > > > we have for dealing with JSON maps, so this might be > really > > easy. > > >>> > Someone > > >>> > > > with more experience in Metron's ES client use might > have a > > better > > >>> idea > > >>> > > of > > >>> > > > the pros and cons to this. The high level client > appears to > > handle > > >>> > > > everything all JSON manipulation for us, but we lose > the > > benefit of > > >>> a > > >>> > > > simpler dependency tree and support for all versions > of ES. > > My only > > >>> > > concern > > >>> > > > with "supports all versions" is that I have to imagine > > there are > > >>> > specific > > >>> > > > calls that we'd have to be careful of when > constructing the > > JSON > > >>> > > requests, > > >>> > > > so it's unclear to me if this is better or worse in > the end. > > >>> > > > > > >>> > > > Best, > > >>> > > > Mike > > >>> > > > > > >>> > > > > > >>> > > > 1. https://www.elastic.co/blog/state-of-the-official- > > >>> > > > elasticsearch-java-clients > > >>> > > > 2. https://www.elastic.co/guide/ > > en/elasticsearch/client/java- > > >>> > > > rest/current/java-rest-high-compatibility.html > > >>> > > > <https://www.elastic.co/guide/ > en/elasticsearch/client/java- > > >>> > > > rest/current/java-rest-high-compatibility.html> > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > On Wed, Sep 27, 2017 at 8:03 PM, Michael Miklavcic < > > >>> > > > michael.miklav...@gmail.com> wrote: > > >>> > > > > > >>> > > > > I am working on upgrading Elasticsearch and Kibana. > There > > are > > >>> quite a > > >>> > > few > > >>> > > > > changes involved with this vix. I believe I'm mostly > > finished with > > >>> > the > > >>> > > > > Ambari mpack side of things, however we currently > only > > support one > > >>> > > > version > > >>> > > > > with no backwards compatibility. What is the > community's > > thoughts > > >>> on > > >>> > > > this? > > >>> > > > > > > >>> > > > > Here is some work contributed to the community that > I'm > > >>> referencing > > >>> > > while > > >>> > > > > working on this upgrade - > https://github.com/apache/ > > >>> > > > metron/pull/619/files > > >>> > > > > > > >>> > > > > Best, > > >>> > > > > Michael Miklavcic > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > > > ------------------- > > Thank you, > > > > James Sirota > > PMC- Apache Metron > > jsirota AT apache DOT org > > > > > > > > > > >