Okay, I have discussed this on <https://lists.apache.org/thread.html/a4e2e7bb7fb7497033696645b011c5604790f23f3802aaab32f1bd01@%3Cdev.metron.apache.org%3E> numerous <https://lists.apache.org/thread.html/1094762a5cf3d92d08ebda416863358fc51d2f96d675f3fc69cefb15@%3Cgeneral.incubator.apache.org%3E> Apache <https://lists.apache.org/thread.html/c92acd125dae05f0537d4505e0254dfa6382ca9f40edba7d2f4c6224@%3Cdev.metron.apache.org%3E> lists <https://lists.apache.org/thread.html/0b669b7460048076b01f87158a1895257223920d8712228c1135bd19@%3Clegal-discuss.apache.org%3E> and it looks like we just need to have the original contributor (Nick) submit the relevant change <https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db065348da0a5043a8353b4a0a8> to our code base <https://github.com/JonZeolla/incubator-metron/tree/METRON-821/metron-sensors/bro-plugin-kafka> via METRON-348 <https://issues.apache.org/jira/browse/METRON-348>. No LICENSE modifications will be needed with this approach, afaict.
After that, I will happily handle everything else, including: 1. A fix for a currently unpatched thread safety issue causing segfaults ( METRON-858 <https://issues.apache.org/jira/browse/METRON-858>) 2. Migration of the kafka plugin to a bro package (METRON-813 <https://issues.apache.org/jira/browse/METRON-813>) - Update of our Ansible playbooks to leverage the package instead of the local code - Removal of bro-plugin-kafka <https://github.com/JonZeolla/incubator-metron/tree/METRON-348/metron-sensors/bro-plugin-kafka> from the incubator-metron repo Jon On Wed, Apr 5, 2017 at 6:01 PM Nick Allen <n...@nickallen.org> wrote: > Created the INFRA request. > > https://issues.apache.org/jira/browse/INFRA-13828 > > > On Wed, Apr 5, 2017 at 5:00 PM, zeo...@gmail.com <zeo...@gmail.com> wrote: > > > Mentor discussion sent > > <https://lists.apache.org/thread.html/a4e2e7bb7fb7497033696645b011c5 > > 604790f23f3802aaab32f1bd01@%3Cdev.metron.apache.org%3E> > > > > Jon > > > > On Wed, Apr 5, 2017 at 4:34 PM zeo...@gmail.com <zeo...@gmail.com> > wrote: > > > > > incubator-metron-bro-plugin-kafka is fine with me, in the hopes that > > when > > > we graduate it becomes metron-bro-plugin-kafka. We could also remove > > > 'plugin' and make it incubator-metron-bro-kafka. > > > > > > Also, I will submit a [MENTORS] discussion. > > > > > > Jon > > > > > > On Wed, Apr 5, 2017 at 4:28 PM Matt Foley <ma...@apache.org> wrote: > > > > > > Browsing https://git.apache.org/ shows lots of examples. A few are > > quite > > > prolific. > > > > > > > > > On 4/5/17, 1:00 PM, "Nick Allen" <n...@nickallen.org> wrote: > > > > > > Does anyone know any other Apache projects that are using multiple > > > repos? > > > I'd like to see what they've done just so we don't break > convention. > > > > > > > > > > > > > > > > > > > > > On Wed, Apr 5, 2017 at 3:22 PM, Nick Allen <n...@nickallen.org> > > wrote: > > > > > > > Yes, I will open an INFRA ticket. Just give me a little time to > > > research > > > > what we need. > > > > > > > > On Wed, Apr 5, 2017 at 1:29 PM, zeo...@gmail.com < > zeo...@gmail.com > > > > > > wrote: > > > > > > > >> Okay great, thanks. Would you mind throwing in an INFRA ticket > > for > > > the > > > >> new > > > >> repo? I can take it all from there. > > > >> > > > >> Does anybody know if we have ASF resources to help answer the > > above > > > legal > > > >> question? > > > >> > > > >> Jon > > > >> > > > >> On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <n...@nickallen.org> > > > wrote: > > > >> > > > >> > (1) I am not sure if licensing is a problem here. > > > >> > > > > >> > (2) I am OK with whatever we need to get this effort done and > > > under ASF. > > > >> > > > > >> > > > > >> > On Wed, Apr 5, 2017 at 1:12 PM, zeo...@gmail.com < > > > zeo...@gmail.com> > > > >> wrote: > > > >> > > > > >> > > I'm working on this > > > >> > > <https://github.com/JonZeolla/incubator-metron/tree/METRON- > > 348> > > > in > > > >> > > preparation for the new repo and migration to a package. It > > > looks > > > >> like > > > >> > in > > > >> > > bro-plugins COPYING > > > >> > > < > https://github.com/bro/bro-plugins/blob/master/kafka/COPYING > > > > > > >> > attributes > > > >> > > to Nick, but in our version COPYING > > > >> > > <https://github.com/apache/incubator-metron/tree/master/ > > > >> > > metron-sensors/bro-plugin-kafka/COPYING> > > > >> > > points to the Apache License. Same with MAINTAINER (this > > > >> > > <https://github.com/apache/incubator-metron/blob/master/ > > > >> > > metron-sensors/bro-plugin-kafka/MAINTAINER> > > > >> > > vs this < > > > https://github.com/bro/bro-plugins/blob/master/kafka/MAINTA > > > >> INER > > > >> > > >). > > > >> > > I assume when we package this up and host it in Apache we > need > > > to > > > >> give it > > > >> > > the Apache license, and point to Metron for MAINTAINER. My > > > questions > > > >> > are: > > > >> > > > > > >> > > 1. Is there any legal/licensing concern here? I am taking > > > changes > > > >> from > > > >> > > the bro-plugins version and pulling it into the > Apache-hosted > > > code. > > > >> > IANAL > > > >> > > 2. Nick - are you OK with these changes? > > > >> > > > > > >> > > Jon > > > >> > > > > > >> > > On Mon, Apr 3, 2017 at 3:50 PM zeo...@gmail.com < > > > zeo...@gmail.com> > > > >> > wrote: > > > >> > > > > > >> > > > Can someone on the PMC submit a ticket to INFRA? It looks > > > like > > > >> > > > <https://www.apache.org/dev/infra-contact> committers > > aren't > > > >> supposed > > > >> > > to. > > > >> > > > > > > >> > > > Jon > > > >> > > > > > > >> > > > On Fri, Mar 31, 2017 at 4:23 PM zeo...@gmail.com < > > > zeo...@gmail.com> > > > >> > > wrote: > > > >> > > > > > > >> > > > I would be happy to try it again but I attempted to do > that > > > before > > > >> with > > > >> > > > bro packages and it failed to be able to handle it. I > also > > > tried > > > >> using > > > >> > > > branches of a repo with bro but that similarly failed (and > > > was a > > > >> pretty > > > >> > > bad > > > >> > > > idea to start with). > > > >> > > > > > > >> > > > Jon > > > >> > > > > > > >> > > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley < > ma...@apache.org> > > > wrote: > > > >> > > > > > > >> > > > We should be able to request just one alternate repo from > > > INFRA, and > > > >> > put > > > >> > > a > > > >> > > > top hierarchical level in it that doesn’t include a maven > > > pom. As > > > >> far > > > >> > as > > > >> > > > maven and clients are concerned, it > > > >> > > > > > > >> > > > just increases by 1 the path length to the root of the > repo. > > > >> > > > > > > >> > > > On 3/31/17, 10:30 AM, "zeo...@gmail.com" < > zeo...@gmail.com> > > > wrote: > > > >> > > > > > > >> > > > Once we agree on a repo location to host this, I would > > be > > > happy > > > >> to > > > >> > > put > > > >> > > > together the package and update our environments to > use > > > bro-pkg > > > >> to > > > >> > > > install > > > >> > > > the plugin. I have created METRON-813 > > > >> > > > <https://issues.apache.org/jira/browse/METRON-813> to > > > track > > > >> this > > > >> > and > > > >> > > > changed METRON-348 < > > > >> > https://issues.apache.org/jira/browse/METRON-348 > > > >> > > > > > > >> > > > to be > > > >> > > > a sub-task. > > > >> > > > > > > >> > > > Otto - the bro packages model doesn't allow colocation > > > with > > > >> > anything > > > >> > > > else. > > > >> > > > That said, if we have two similar situations, and > given > > > the > > > >> INFRA > > > >> > > > example > > > >> > > > <https://issues.apache.org/jira/browse/INFRA-7060> > > Casey > > > >> linked to > > > >> > > > before > > > >> > > > was requesting 9 repos, perhaps we just request two > > repos. > > > >> Would > > > >> > > > someone > > > >> > > > else mind putting that request in? > > > >> > > > > > > >> > > > Jon > > > >> > > > > > > >> > > > On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler < > > > >> > > ottobackwa...@gmail.com> > > > >> > > > wrote: > > > >> > > > > > > >> > > > Could we create a separate repo for more than on > thing? > > > like > > > >> put … > > > >> > > um > > > >> > > > let’s say > > > >> > > > a maven plugin and the bro plugin? > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > On March 31, 2017 at 12:30:25, Nick Allen ( > > > n...@nickallen.org) > > > >> > > wrote: > > > >> > > > > > > >> > > > I agree with everything that I've read. > > > >> > > > > > > >> > > > One of the guys from Bro had contacted me a while > back, > > > letting > > > >> me > > > >> > > know > > > >> > > > that the packaging mechanism in Bro was ready for > public > > > >> > > consumption. I > > > >> > > > just have not had cycles to do anything with it yet. > > They > > > are > > > >> not > > > >> > > > wanting > > > >> > > > to host any of the plugins. > > > >> > > > > > > >> > > > I thought the package mechanism requires that a > package > > > live > > > >> within > > > >> > > > its own > > > >> > > > repo (which Casey confirmed). This put me in a bind on > > > how to > > > >> > tackle > > > >> > > > this. I don't want to personally host the plugin in my > > own > > > >> Github > > > >> > > > repo. I > > > >> > > > would prefer that we host it in a community repo; > either > > > Bro or > > > >> > > Metron. > > > >> > > > Since Bro is moving away from hosting their own > plugins, > > > that > > > >> > leaves > > > >> > > > Metron. > > > >> > > > > > > >> > > > It would be great if we could create a separate repo > for > > > the > > > >> > plugin. > > > >> > > > That > > > >> > > > solves the challenge of using the packaging mechanism. > > > >> > > > > > > >> > > > We do need to reconcile what is in bro/bro-plugins and > > > what is > > > >> in > > > >> > > > Metron. > > > >> > > > There are some enhancements that I and others have > made > > > that > > > >> never > > > >> > > > made it > > > >> > > > back into Metron. They never made it back, because the > > > original > > > >> > plan > > > >> > > > was > > > >> > > > just to switch to using the plugin from > bro/bro-plugins > > > before > > > >> the > > > >> > > > idea of > > > >> > > > a packaging mechanism hit Bro. Reconciling should be > > > fairly > > > >> easy to > > > >> > > > see by > > > >> > > > just doing a diff. > > > >> > > > > > > >> > > > It would be great if others want to take on any of > that > > > work. I > > > >> > would > > > >> > > > be > > > >> > > > glad to offer any support that you need. Thanks, Jon! > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > On Thu, Mar 30, 2017 at 11:20 PM, zeo...@gmail.com < > > > >> > zeo...@gmail.com > > > >> > > > > > > >> > > > wrote: > > > >> > > > > > > >> > > > > Ok, great. > > > >> > > > > > > > >> > > > > I agree, I definitely want to hear from Nick on the > > > topic. My > > > >> > team > > > >> > > is > > > >> > > > > currently looking into enhancing the plugin as well > to > > > >> > potentially > > > >> > > > allow > > > >> > > > > sending to multiple clusters, investigating some > > issues > > > we see > > > >> > when > > > >> > > > our > > > >> > > > bro > > > >> > > > > cluster is under load, turn it into a package, etc. > > > >> > > > > > > > >> > > > > The work you just did was on our to do list as well > so > > > I'm > > > >> very > > > >> > > > excited > > > >> > > > to > > > >> > > > > see it come through. > > > >> > > > > > > > >> > > > > Jon > > > >> > > > > > > > >> > > > > On Thu, Mar 30, 2017, 11:16 PM Casey Stella < > > > >> ceste...@gmail.com> > > > >> > > > wrote: > > > >> > > > > > > > >> > > > > I *think* it's possible. People do ask for mirrors > of > > > >> directories > > > >> > > > from > > > >> > > > > time to time (see > > > >> > https://issues.apache.org/jira/browse/INFRA-7060 > > > >> > > ). > > > >> > > > If > > > >> > > > > we > > > >> > > > > think this is a good idea, we can pose it to INFRA > as > > a > > > >> request. > > > >> > > I'd > > > >> > > > love > > > >> > > > > to see us be able to use the bro packaging > > > infrastructure and > > > >> get > > > >> > > > more > > > >> > > > > visibility for the plugin. > > > >> > > > > > > > >> > > > > I'd be particularly interested in Nick's opinion on > > > this, > > > >> though. > > > >> > > > > > > > >> > > > > On Thu, Mar 30, 2017 at 11:12 PM, zeo...@gmail.com > < > > > >> > > zeo...@gmail.com > > > >> > > > > > > > >> > > > > wrote: > > > >> > > > > > > > >> > > > > > You can version packages - > > > >> > > > > > http://bro-package-manager. > > readthedocs.io/en/stable/ > > > >> > > > > package.html#package- > > > >> > > > > > versioning > > > >> > > > > > > > > >> > > > > > I agree that having a separate repo provided by > > Apache > > > >> would be > > > >> > > > optimal, > > > >> > > > > I > > > >> > > > > > just don't know the process for that or if it was > > even > > > >> > reasonable > > > >> > > > to > > > >> > > > > > suggest. > > > >> > > > > > > > > >> > > > > > Jon > > > >> > > > > > > > > >> > > > > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella < > > > >> > ceste...@gmail.com> > > > >> > > > wrote: > > > >> > > > > > > > > >> > > > > > > Looking at the bro packages, it appears that bro > > is > > > >> expecting > > > >> > > > things > > > >> > > > to > > > >> > > > > > be > > > >> > > > > > > its own git repository. I wonder if we could > > either > > > >> request > > > >> > > INFRA > > > >> > > > > > provide > > > >> > > > > > > another repo for the bro-kafka plugin and > > integrate > > > it > > > >> into > > > >> > > > metron as > > > >> > > > a > > > >> > > > > > git > > > >> > > > > > > submodule *or* if we could request INFRA to > > create a > > > >> github > > > >> > > > mirror of > > > >> > > > > the > > > >> > > > > > > metron-sensors/bro-kafka-plugin directory. I'm > > not > > > sure > > > >> how > > > >> > > > viable > > > >> > > > > > either > > > >> > > > > > > of those options are, frankly. > > > >> > > > > > > > > > >> > > > > > > One thing that I didn't see is how do you > specify > > a > > > >> > particular > > > >> > > > release > > > >> > > > > of > > > >> > > > > > > the plugin that you want to install? For us, > we'd > > > want to > > > >> > > > release the > > > >> > > > > > > plugin along with the product. I didn't quite > see > > > how > > > >> you'd > > > >> > > push > > > >> > > > > > releases > > > >> > > > > > > for bro plugins. > > > >> > > > > > > > > > >> > > > > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella < > > > >> > > > ceste...@gmail.com> > > > >> > > > > > wrote: > > > >> > > > > > > > > > >> > > > > > > > So, I do agree with the concern. Is there a > way > > > to host > > > >> the > > > >> > > > package > > > >> > > > > > > > within Metron? I definitely would like to see > > the > > > >> > > > modifications at > > > >> > > > > > > > https://github.com/bro/bro-plu > > > >> gins/commit/b9f1f35415cb0db > > > >> > > > > > > > 065348da0a5043a8353b4a0a8 brought back into > > > Metron and > > > >> I'd > > > >> > > > love for > > > >> > > > > us > > > >> > > > > > to > > > >> > > > > > > > host the plugin. > > > >> > > > > > > > > > > >> > > > > > > > Thoughts? > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > On Thu, Mar 30, 2017 at 9:09 PM, > > zeo...@gmail.com > > > < > > > >> > > > zeo...@gmail.com> > > > >> > > > > > > >> > > > > > > > wrote: > > > >> > > > > > > > > > > >> > > > > > > >> Today I was taking a look at METRON-812 > > > >> > > > > > > >> < > > > https://issues.apache.org/jira/browse/METRON-812>, > > > >> which > > > >> > > > made me > > > >> > > > > > > recall > > > >> > > > > > > >> some conversations from a while back > regarding > > > where > > > >> the > > > >> > bro > > > >> > > > kafka > > > >> > > > > > > plugin > > > >> > > > > > > >> should ultimately live, and how to update it. > > > >> > > > > > > >> > > > >> > > > > > > >> Back in METRON-348 < > https://issues.apache.org/ > > > >> > > > > jira/browse/METRON-348> > > > >> > > > > > I > > > >> > > > > > > >> brought up the fact that some important > changes > > > >> > > > > > > >> <https://github.com/bro/bro-plugins/commit/ > > > >> > > b9f1f35415cb0db06 > > > >> > > > > > > >> 5348da0a5043a8353b4a0a8> > > > >> > > > > > > >> were made to the externally hosted version of > > > the kafka > > > >> > > > plugin, > > > >> > > > and > > > >> > > > > > were > > > >> > > > > > > >> never introduced to Metron's hosted version > > > (i.e. the > > > >> one > > > >> > we > > > >> > > > use > > > >> > > > > > > >> <https://github.com/apache/inc > > > >> ubator-metron/blob/master/ > > > >> > > metr > > > >> > > > > > > >> > > > on-deployment/roles/bro/tasks/bro-plugin-kafka.yml> > > > >> > > > > > > >> in vagrant when bro is installed). The > > > conversation > > > >> went > > > >> > > down > > > >> > > > the > > > >> > > > > > route > > > >> > > > > > > >> of > > > >> > > > > > > >> discussing whether or not the bro kafka > plugin > > > code > > > >> should > > > >> > > > continue > > > >> > > > > to > > > >> > > > > > > >> live > > > >> > > > > > > >> in Metron in the first place. Now, with > > > METRON-812, I > > > >> see > > > >> > us > > > >> > > > > further > > > >> > > > > > > >> muddying the waters of where to go for the > > right > > > >> plugin, > > > >> > as > > > >> > > > our > > > >> > > > > > version > > > >> > > > > > > is > > > >> > > > > > > >> still missing the public changes but adds > some > > > very > > > >> > > important > > > >> > > > new > > > >> > > > > > > >> functionality. > > > >> > > > > > > >> > > > >> > > > > > > >> I'd like to bring up the idea of using bro's > > > packages > > > >> > > > > > > >> <https://github.com/bro/packages> framework, > > > released > > > >> in > > > >> > > > late 2016 > > > >> > > > > > > >> < > > > >> > > > > > > http://blog.bro.org/2016/10/introducing-bro-package-manager.html> > > > >> > > > > > > >> (additional > > > >> > > > > > > >> documentation here < > > > >> > > > > > > > > > http://bro-package-manager.readthedocs.io/en/stable/ > > > >> > > > > > > >> >), > > > >> > > > > > > >> as a potential place for this to be > > > hosted/referenced. > > > >> > This > > > >> > > > is a > > > >> > > > > > simple > > > >> > > > > > > >> and supported method (funded by Mozilla > > > >> > > > > > > >> <https://blog.mozilla.org/blog > > > >> /2015/12/10/mozilla-open- > > > >> > > sourc > > > >> > > > > > > >> e-support-first-awards-made/>) > > > >> > > > > > > >> to install and uninstall bro scripts, > plugins, > > > etc., > > > >> and > > > >> > it > > > >> > > > also > > > >> > > > > > allows > > > >> > > > > > > us > > > >> > > > > > > >> to continue to have enough control over > updates > > > to the > > > >> > > plugin > > > >> > > > so > > > >> > > > > that > > > >> > > > > > it > > > >> > > > > > > >> will not slow down Metron development by > having > > > it as a > > > >> > > > dependency > > > >> > > > > > > >> (resolving both of Casey's concerns noted > here > > > >> > > > > > > >> < > > > https://issues.apache.org/jira/browse/METRON-348? > > > >> > > focusedCom > > > >> > > > > > > >> > > > mentId=15391865&page=com.atlassian.jira.plugin.system. > > > >> > > > > > > >> > > > issuetabpanels:comment-tabpanel#comment-15391865>, > > > >> > > > > > > >> and I think this solution is supported by > > Nick's > > > >> comments > > > >> > > here > > > >> > > > > > > >> < > > > https://issues.apache.org/jira/browse/METRON-348? > > > >> > > focusedCom > > > >> > > > > > > >> > > > mentId=15391872&page=com.atlassian.jira.plugin.system. > > > >> > > > > > > >> issuetabpanels:comment- > > tabpanel#comment-15391872> > > > >> > > > > > > >> as > > > >> > > > > > > >> well). > > > >> > > > > > > >> > > > >> > > > > > > >> The only thing I'm not sure about is where to > > > host the > > > >> > > plugin > > > >> > > > itself > > > >> > > > > - > > > >> > > > > > > my > > > >> > > > > > > >> first thought would be Nick's github < > > > >> > > > https://github.com/nickwallen > > > >> > > > > >, > > > >> > > > > > > as > > > >> > > > > > > >> he > > > >> > > > > > > >> really kicked off this effort, but maybe we > can > > > think > > > >> of > > > >> > > > something > > > >> > > > > > > better. > > > >> > > > > > > >> > > > >> > > > > > > >> Is this approach of interest to anybody? It > is > > > >> extremely > > > >> > > > simple to > > > >> > > > > > put > > > >> > > > > > > >> together - I was able to throw one together > > > >> > > > > > > >> < > https://github.com/bro/packages/blob/master/ > > > >> > > > > jonzeolla/bro-pkg.index> > > > >> > > > > > > and > > > >> > > > > > > >> get it working with a fresh bro 2.5 install > > when > > > >> attending > > > >> > > the > > > >> > > > > brocon > > > >> > > > > > > talk > > > >> > > > > > > >> < > > > >> > https://www.bro.org/brocon2016/brocon2016_abstracts.html# > > > >> > > > > > > >> bro-packagemanager> > > > >> > > > > > > >> (recording <https://www.youtube.com/watch > > > >> ?v=9RFfPJeGkcE>, > > > >> > > > slides > > > >> > > > > > > >> < > > > https://www.bro.org/brocon2016/slides/hall_bpm.pdf>) > > > >> > that > > > >> > > > > introduced > > > >> > > > > > > >> this > > > >> > > > > > > >> to me in the first place. > > > >> > > > > > > >> > > > >> > > > > > > >> Jon > > > >> > > > > > > >> -- > > > >> > > > > > > >> > > > >> > > > > > > >> Jon > > > >> > > > > > > >> > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > -- > > > >> > > > > > > > > >> > > > > > Jon > > > >> > > > > > > > > >> > > > > > > > >> > > > > -- > > > >> > > > > > > > >> > > > > Jon > > > >> > > > > > > > >> > > > > > > >> > > > -- > > > >> > > > > > > >> > > > Jon > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > -- > > > >> > > > > > > >> > > > Jon > > > >> > > > > > > >> > > > -- > > > >> > > > > > > >> > > > Jon > > > >> > > > > > > >> > > -- > > > >> > > > > > >> > > Jon > > > >> > > > > > >> > > > > >> -- > > > >> > > > >> Jon > > > >> > > > > > > > > > > > > > > > > > > > > -- > > > > > > Jon > > > > > -- > > > > Jon > > > -- Jon
