My assumption with HDP 3 was it would include Kafka 0.11 so we could leverage the new idempotent producer + transactions API.
On the bro side, unless you know something I don't the Kafka plugin package is the one I'm working on getting released using the Metron code ( https://github.com/jonzeolla/metron-bro-plugin-kafka which will hopefully soon go to apache/). Packages are new to 2.5, which is one of the reasons I look to move Metron to it, among many others (including some pretty big bugs that have been fixed and performance improvements). I would contribute to a wish list/feature prioritization thread. Jon Sorry for the brevity, writing this on my mobile device. On Wed, Aug 23, 2017, 06:21 Simon Elliston Ball <si...@simonellistonball.com> wrote: > Hi Jon, > > Good points all. Some of the versions we have been using are certainly > aging. There are some PRs and tickets out for ES 5 upgrade, and some fine > work has already been done on Centos 7 installation. I think it would be > well worth us making some community decisions about bumping up some of the > default versions. > > There is also a strong reason to bump up the bro version. My understanding > is that the latest bro has now incorporated something similar to our kafka > plugin, so maybe with the new bro, we can deprecate our plugin and just use > the upstream to reduce the maintenance burden. > > Personally I run my clusters on HDP 2.6.1 today, but this is certainly not > fully tested and the project is still very much HDP 2.5 (full dev etc are > there). So, it works… we should look more carefully at that statement than > I have though! > > It would be good to hear a little more about your thoughts on exactly > once. Not sure HDP 3 is going to help us much there. > > On the general feature direction and requests, it would be great to hear > from everyone on thoughts for future direction and things they might want > to see in the project. Perhaps we should have a discuss thread to capture > wish lists. > > Thoughts? > > Simon > > > On 23 Aug 2017, at 11:08, zeo...@gmail.com <zeo...@gmail.com> wrote: > > > > Was there any discussion about future features of Metron aside from > > 777/942? In the initial announce thread the agenda mentioned where want > to > > take the project long-term and feature requests and comments on existing > > features. > > > > My thoughts on the topic are that I would like to see a move quickly > after > > 0.4.1 to upgrade the stack, from centos 7, to elasticsearch 5, bro 2.5.1, > > HDP 2.6.1, etc. There's a growing list of issues due to the older > > platforms we use. I have already started work on some of this, and I > know > > we have a PR open for an ES upgrade as well. > > > > More forward looking, I would also like to see a push for exactly once > > processing through the stack, which I expect will be available with HDP > > 3.x? > > > > I wholeheartedly agree that we should get back on top of the community > > demos. They have been helpful for me to watch and are very good > materials > > for pointing people new to the project to, in addition to our site-book. > > > > Jon > > > > On Tue, Aug 22, 2017, 15:00 Casey Stella <ceste...@gmail.com> wrote: > > > >> Introduction by James Sirota and hand off to me. > >> > >> Topic: Apache Administrivia > >> > >> - We have passed our 3 month window of submitting board reports > without > >> any serious incident > >> - We need and want more committers. Anyone who is so inclined and > >> interested should pitch in, even without code contributions > >> > >> Topic: Release > >> > >> - General consensus that a release is in order > >> - The release should be a point release and pre-777 > >> > >> Topic: The Next Steps for METRON-777 > >> > >> - Proposed: We should move METRON-777 and METRON-942 into a feature > >> branch > >> - Proposed: Once working sufficiently, the first milestone is for Otto > >> to create a youtube video showing off his work and have a discuss > thread > >> for a second pass of architecture/feature review > >> - Proposed: This feature branch should have no less stringent > >> requirements than we have for master > >> - Proposed: Documentation, testing, and the normal accompanying > >> artifacts for any other feature should be in place before we merge. > >> Other > >> more feature-specific criteria should be discussed in separate discuss > >> threads. > >> - No dissenting opinions on the call. Otto to create a discuss thread > >> about the feature branch and the mechanics of actually creating it. > >> > >> Topic: Community Demos > >> > >> - Proposed: We should have community demos again on a monthly basis. > >> - General assent to this as well. > >> - James to set up a webex for one month hence. > >> > >> Anything that I got wrong or missed, please respond to this discuss > thread. > >> > >> Best, > >> > >> Casey > >> > > -- > > > > Jon > > -- Jon
