fine by me. On Wed, Apr 5, 2017 at 4:16 PM, James Sirota <jsir...@apache.org> wrote:
> I don't mind incubator-metron-bro-plugin-kafka > > 05.04.2017, 13:15, "Nick Allen" <n...@nickallen.org>: > > Ok, I found that CouchDB and Nifi, among others, use multiple repos. A > > fairly obvious pattern seems to emerge. :) > > > > git://git.apache.org/couchdb-fabric.git > > git://git.apache.org/couchdb-couch-replicator.git > > git://git.apache.org/couchdb-chttpd.git > > > > git://git.apache.org/nifi > > git://git.apache.org/nifi-site.git > > > > > > > > So what shall we call the new repo? Using > > "incubator-metron-bro-plugin-kafka" is the obvious choice, but seems > > excessively long. Totally open to better name for the plugin itself. > > > > On Wed, Apr 5, 2017 at 4:00 PM, Nick Allen <n...@nickallen.org> wrote: > > > >> Does anyone know any other Apache projects that are using multiple > repos? > >> I'd like to see what they've done just so we don't break convention. > >> > >> On Wed, Apr 5, 2017 at 3:22 PM, Nick Allen <n...@nickallen.org> wrote: > >> > >>> Yes, I will open an INFRA ticket. Just give me a little time to > research > >>> what we need. > >>> > >>> On Wed, Apr 5, 2017 at 1:29 PM, zeo...@gmail.com <zeo...@gmail.com> > >>> wrote: > >>> > >>>> Okay great, thanks. Would you mind throwing in an INFRA ticket for > the > >>>> new > >>>> repo? I can take it all from there. > >>>> > >>>> Does anybody know if we have ASF resources to help answer the above > legal > >>>> question? > >>>> > >>>> Jon > >>>> > >>>> On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <n...@nickallen.org> > wrote: > >>>> > >>>> > (1) I am not sure if licensing is a problem here. > >>>> > > >>>> > (2) I am OK with whatever we need to get this effort done and under > >>>> ASF. > >>>> > > >>>> > > >>>> > On Wed, Apr 5, 2017 at 1:12 PM, zeo...@gmail.com <zeo...@gmail.com > > > >>>> wrote: > >>>> > > >>>> > > I'm working on this > >>>> > > <https://github.com/JonZeolla/incubator-metron/tree/METRON-348> > in > >>>> > > preparation for the new repo and migration to a package. It looks > >>>> like > >>>> > in > >>>> > > bro-plugins COPYING > >>>> > > <https://github.com/bro/bro-plugins/blob/master/kafka/COPYING> > >>>> > attributes > >>>> > > to Nick, but in our version COPYING > >>>> > > <https://github.com/apache/incubator-metron/tree/master/ > >>>> > > metron-sensors/bro-plugin-kafka/COPYING> > >>>> > > points to the Apache License. Same with MAINTAINER (this > >>>> > > <https://github.com/apache/incubator-metron/blob/master/ > >>>> > > metron-sensors/bro-plugin-kafka/MAINTAINER> > >>>> > > vs this <https://github.com/bro/bro-plugins/blob/master/kafka/ > MAINTA > >>>> INER > >>>> > > >). > >>>> > > I assume when we package this up and host it in Apache we need to > >>>> give it > >>>> > > the Apache license, and point to Metron for MAINTAINER. My > questions > >>>> > are: > >>>> > > > >>>> > > 1. Is there any legal/licensing concern here? I am taking changes > >>>> from > >>>> > > the bro-plugins version and pulling it into the Apache-hosted > code. > >>>> > IANAL > >>>> > > 2. Nick - are you OK with these changes? > >>>> > > > >>>> > > Jon > >>>> > > > >>>> > > On Mon, Apr 3, 2017 at 3:50 PM zeo...@gmail.com < > zeo...@gmail.com> > >>>> > wrote: > >>>> > > > >>>> > > > Can someone on the PMC submit a ticket to INFRA? It looks like > >>>> > > > <https://www.apache.org/dev/infra-contact> committers aren't > >>>> supposed > >>>> > > to. > >>>> > > > > >>>> > > > Jon > >>>> > > > > >>>> > > > On Fri, Mar 31, 2017 at 4:23 PM zeo...@gmail.com < > zeo...@gmail.com > >>>> > > >>>> > > wrote: > >>>> > > > > >>>> > > > I would be happy to try it again but I attempted to do that > before > >>>> with > >>>> > > > bro packages and it failed to be able to handle it. I also > tried > >>>> using > >>>> > > > branches of a repo with bro but that similarly failed (and was > a > >>>> pretty > >>>> > > bad > >>>> > > > idea to start with). > >>>> > > > > >>>> > > > Jon > >>>> > > > > >>>> > > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <ma...@apache.org> > wrote: > >>>> > > > > >>>> > > > We should be able to request just one alternate repo from > INFRA, > >>>> and > >>>> > put > >>>> > > a > >>>> > > > top hierarchical level in it that doesn’t include a maven pom. > As > >>>> far > >>>> > as > >>>> > > > maven and clients are concerned, it > >>>> > > > > >>>> > > > just increases by 1 the path length to the root of the repo. > >>>> > > > > >>>> > > > On 3/31/17, 10:30 AM, "zeo...@gmail.com" <zeo...@gmail.com> > wrote: > >>>> > > > > >>>> > > > Once we agree on a repo location to host this, I would be > >>>> happy to > >>>> > > put > >>>> > > > together the package and update our environments to use > >>>> bro-pkg to > >>>> > > > install > >>>> > > > the plugin. I have created METRON-813 > >>>> > > > <https://issues.apache.org/jira/browse/METRON-813> to track > >>>> this > >>>> > and > >>>> > > > changed METRON-348 < > >>>> > https://issues.apache.org/jira/browse/METRON-348 > >>>> > > > > >>>> > > > to be > >>>> > > > a sub-task. > >>>> > > > > >>>> > > > Otto - the bro packages model doesn't allow colocation with > >>>> > anything > >>>> > > > else. > >>>> > > > That said, if we have two similar situations, and given the > >>>> INFRA > >>>> > > > example > >>>> > > > <https://issues.apache.org/jira/browse/INFRA-7060> Casey > >>>> linked to > >>>> > > > before > >>>> > > > was requesting 9 repos, perhaps we just request two repos. > >>>> Would > >>>> > > > someone > >>>> > > > else mind putting that request in? > >>>> > > > > >>>> > > > Jon > >>>> > > > > >>>> > > > On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler < > >>>> > > ottobackwa...@gmail.com> > >>>> > > > wrote: > >>>> > > > > >>>> > > > Could we create a separate repo for more than on thing? like > >>>> put … > >>>> > > um > >>>> > > > let’s say > >>>> > > > a maven plugin and the bro plugin? > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > On March 31, 2017 at 12:30:25, Nick Allen (n...@nickallen.org) > >>>> > > wrote: > >>>> > > > > >>>> > > > I agree with everything that I've read. > >>>> > > > > >>>> > > > One of the guys from Bro had contacted me a while back, > >>>> letting me > >>>> > > know > >>>> > > > that the packaging mechanism in Bro was ready for public > >>>> > > consumption. I > >>>> > > > just have not had cycles to do anything with it yet. They are > >>>> not > >>>> > > > wanting > >>>> > > > to host any of the plugins. > >>>> > > > > >>>> > > > I thought the package mechanism requires that a package live > >>>> within > >>>> > > > its own > >>>> > > > repo (which Casey confirmed). This put me in a bind on how to > >>>> > tackle > >>>> > > > this. I don't want to personally host the plugin in my own > >>>> Github > >>>> > > > repo. I > >>>> > > > would prefer that we host it in a community repo; either Bro or > >>>> > > Metron. > >>>> > > > Since Bro is moving away from hosting their own plugins, that > >>>> > leaves > >>>> > > > Metron. > >>>> > > > > >>>> > > > It would be great if we could create a separate repo for the > >>>> > plugin. > >>>> > > > That > >>>> > > > solves the challenge of using the packaging mechanism. > >>>> > > > > >>>> > > > We do need to reconcile what is in bro/bro-plugins and what is > >>>> in > >>>> > > > Metron. > >>>> > > > There are some enhancements that I and others have made that > >>>> never > >>>> > > > made it > >>>> > > > back into Metron. They never made it back, because the original > >>>> > plan > >>>> > > > was > >>>> > > > just to switch to using the plugin from bro/bro-plugins before > >>>> the > >>>> > > > idea of > >>>> > > > a packaging mechanism hit Bro. Reconciling should be fairly > >>>> easy to > >>>> > > > see by > >>>> > > > just doing a diff. > >>>> > > > > >>>> > > > It would be great if others want to take on any of that work. I > >>>> > would > >>>> > > > be > >>>> > > > glad to offer any support that you need. Thanks, Jon! > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > On Thu, Mar 30, 2017 at 11:20 PM, zeo...@gmail.com < > >>>> > zeo...@gmail.com > >>>> > > > > >>>> > > > wrote: > >>>> > > > > >>>> > > > > Ok, great. > >>>> > > > > > >>>> > > > > I agree, I definitely want to hear from Nick on the topic. My > >>>> > team > >>>> > > is > >>>> > > > > currently looking into enhancing the plugin as well to > >>>> > potentially > >>>> > > > allow > >>>> > > > > sending to multiple clusters, investigating some issues we > >>>> see > >>>> > when > >>>> > > > our > >>>> > > > bro > >>>> > > > > cluster is under load, turn it into a package, etc. > >>>> > > > > > >>>> > > > > The work you just did was on our to do list as well so I'm > >>>> very > >>>> > > > excited > >>>> > > > to > >>>> > > > > see it come through. > >>>> > > > > > >>>> > > > > Jon > >>>> > > > > > >>>> > > > > On Thu, Mar 30, 2017, 11:16 PM Casey Stella < > >>>> ceste...@gmail.com> > >>>> > > > wrote: > >>>> > > > > > >>>> > > > > I *think* it's possible. People do ask for mirrors of > >>>> directories > >>>> > > > from > >>>> > > > > time to time (see > >>>> > https://issues.apache.org/jira/browse/INFRA-7060 > >>>> > > ). > >>>> > > > If > >>>> > > > > we > >>>> > > > > think this is a good idea, we can pose it to INFRA as a > >>>> request. > >>>> > > I'd > >>>> > > > love > >>>> > > > > to see us be able to use the bro packaging infrastructure > >>>> and get > >>>> > > > more > >>>> > > > > visibility for the plugin. > >>>> > > > > > >>>> > > > > I'd be particularly interested in Nick's opinion on this, > >>>> though. > >>>> > > > > > >>>> > > > > On Thu, Mar 30, 2017 at 11:12 PM, zeo...@gmail.com < > >>>> > > zeo...@gmail.com > >>>> > > > > > >>>> > > > > wrote: > >>>> > > > > > >>>> > > > > > You can version packages - > >>>> > > > > > http://bro-package-manager.readthedocs.io/en/stable/ > >>>> > > > > package.html#package- > >>>> > > > > > versioning > >>>> > > > > > > >>>> > > > > > I agree that having a separate repo provided by Apache > >>>> would be > >>>> > > > optimal, > >>>> > > > > I > >>>> > > > > > just don't know the process for that or if it was even > >>>> > reasonable > >>>> > > > to > >>>> > > > > > suggest. > >>>> > > > > > > >>>> > > > > > Jon > >>>> > > > > > > >>>> > > > > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella < > >>>> > ceste...@gmail.com> > >>>> > > > wrote: > >>>> > > > > > > >>>> > > > > > > Looking at the bro packages, it appears that bro is > >>>> expecting > >>>> > > > things > >>>> > > > to > >>>> > > > > > be > >>>> > > > > > > its own git repository. I wonder if we could either > >>>> request > >>>> > > INFRA > >>>> > > > > > provide > >>>> > > > > > > another repo for the bro-kafka plugin and integrate it > >>>> into > >>>> > > > metron as > >>>> > > > a > >>>> > > > > > git > >>>> > > > > > > submodule *or* if we could request INFRA to create a > >>>> github > >>>> > > > mirror of > >>>> > > > > the > >>>> > > > > > > metron-sensors/bro-kafka-plugin directory. I'm not sure > >>>> how > >>>> > > > viable > >>>> > > > > > either > >>>> > > > > > > of those options are, frankly. > >>>> > > > > > > > >>>> > > > > > > One thing that I didn't see is how do you specify a > >>>> > particular > >>>> > > > release > >>>> > > > > of > >>>> > > > > > > the plugin that you want to install? For us, we'd want to > >>>> > > > release the > >>>> > > > > > > plugin along with the product. I didn't quite see how > >>>> you'd > >>>> > > push > >>>> > > > > > releases > >>>> > > > > > > for bro plugins. > >>>> > > > > > > > >>>> > > > > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella < > >>>> > > > ceste...@gmail.com> > >>>> > > > > > wrote: > >>>> > > > > > > > >>>> > > > > > > > So, I do agree with the concern. Is there a way to > >>>> host the > >>>> > > > package > >>>> > > > > > > > within Metron? I definitely would like to see the > >>>> > > > modifications at > >>>> > > > > > > > https://github.com/bro/bro-plu > >>>> gins/commit/b9f1f35415cb0db > >>>> > > > > > > > 065348da0a5043a8353b4a0a8 brought back into Metron and > >>>> I'd > >>>> > > > love for > >>>> > > > > us > >>>> > > > > > to > >>>> > > > > > > > host the plugin. > >>>> > > > > > > > > >>>> > > > > > > > Thoughts? > >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > On Thu, Mar 30, 2017 at 9:09 PM, zeo...@gmail.com < > >>>> > > > zeo...@gmail.com> > >>>> > > > > >>>> > > > > > > > wrote: > >>>> > > > > > > > > >>>> > > > > > > >> Today I was taking a look at METRON-812 > >>>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-812>, > >>>> which > >>>> > > > made me > >>>> > > > > > > recall > >>>> > > > > > > >> some conversations from a while back regarding where > >>>> the > >>>> > bro > >>>> > > > kafka > >>>> > > > > > > plugin > >>>> > > > > > > >> should ultimately live, and how to update it. > >>>> > > > > > > >> > >>>> > > > > > > >> Back in METRON-348 <https://issues.apache.org/ > >>>> > > > > jira/browse/METRON-348> > >>>> > > > > > I > >>>> > > > > > > >> brought up the fact that some important changes > >>>> > > > > > > >> <https://github.com/bro/bro-plugins/commit/ > >>>> > > b9f1f35415cb0db06 > >>>> > > > > > > >> 5348da0a5043a8353b4a0a8> > >>>> > > > > > > >> were made to the externally hosted version of the > >>>> kafka > >>>> > > > plugin, > >>>> > > > and > >>>> > > > > > were > >>>> > > > > > > >> never introduced to Metron's hosted version (i.e. the > >>>> one > >>>> > we > >>>> > > > use > >>>> > > > > > > >> <https://github.com/apache/inc > >>>> ubator-metron/blob/master/ > >>>> > > metr > >>>> > > > > > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml> > >>>> > > > > > > >> in vagrant when bro is installed). The conversation > >>>> went > >>>> > > down > >>>> > > > the > >>>> > > > > > route > >>>> > > > > > > >> of > >>>> > > > > > > >> discussing whether or not the bro kafka plugin code > >>>> should > >>>> > > > continue > >>>> > > > > to > >>>> > > > > > > >> live > >>>> > > > > > > >> in Metron in the first place. Now, with METRON-812, I > >>>> see > >>>> > us > >>>> > > > > further > >>>> > > > > > > >> muddying the waters of where to go for the right > >>>> plugin, > >>>> > as > >>>> > > > our > >>>> > > > > > version > >>>> > > > > > > is > >>>> > > > > > > >> still missing the public changes but adds some very > >>>> > > important > >>>> > > > new > >>>> > > > > > > >> functionality. > >>>> > > > > > > >> > >>>> > > > > > > >> I'd like to bring up the idea of using bro's packages > >>>> > > > > > > >> <https://github.com/bro/packages> framework, > >>>> released in > >>>> > > > late 2016 > >>>> > > > > > > >> < > >>>> > > > http://blog.bro.org/2016/10/introducing-bro-package- > manager.html> > >>>> > > > > > > >> (additional > >>>> > > > > > > >> documentation here < > >>>> > > > > > > http://bro-package-manager.readthedocs.io/en/stable/ > >>>> > > > > > > >> >), > >>>> > > > > > > >> as a potential place for this to be hosted/referenced. > >>>> > This > >>>> > > > is a > >>>> > > > > > simple > >>>> > > > > > > >> and supported method (funded by Mozilla > >>>> > > > > > > >> <https://blog.mozilla.org/blog > >>>> /2015/12/10/mozilla-open- > >>>> > > sourc > >>>> > > > > > > >> e-support-first-awards-made/>) > >>>> > > > > > > >> to install and uninstall bro scripts, plugins, etc., > >>>> and > >>>> > it > >>>> > > > also > >>>> > > > > > allows > >>>> > > > > > > us > >>>> > > > > > > >> to continue to have enough control over updates to the > >>>> > > plugin > >>>> > > > so > >>>> > > > > that > >>>> > > > > > it > >>>> > > > > > > >> will not slow down Metron development by having it as > >>>> a > >>>> > > > dependency > >>>> > > > > > > >> (resolving both of Casey's concerns noted here > >>>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348? > >>>> > > focusedCom > >>>> > > > > > > >> mentId=15391865&page=com.atlas > >>>> sian.jira.plugin.system. > >>>> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391865>, > >>>> > > > > > > >> and I think this solution is supported by Nick's > >>>> comments > >>>> > > here > >>>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348? > >>>> > > focusedCom > >>>> > > > > > > >> mentId=15391872&page=com.atlas > >>>> sian.jira.plugin.system. > >>>> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391872> > >>>> > > > > > > >> as > >>>> > > > > > > >> well). > >>>> > > > > > > >> > >>>> > > > > > > >> The only thing I'm not sure about is where to host the > >>>> > > plugin > >>>> > > > itself > >>>> > > > > - > >>>> > > > > > > my > >>>> > > > > > > >> first thought would be Nick's github < > >>>> > > > https://github.com/nickwallen > >>>> > > > > >, > >>>> > > > > > > as > >>>> > > > > > > >> he > >>>> > > > > > > >> really kicked off this effort, but maybe we can think > >>>> of > >>>> > > > something > >>>> > > > > > > better. > >>>> > > > > > > >> > >>>> > > > > > > >> Is this approach of interest to anybody? It is > >>>> extremely > >>>> > > > simple to > >>>> > > > > > put > >>>> > > > > > > >> together - I was able to throw one together > >>>> > > > > > > >> <https://github.com/bro/packages/blob/master/ > >>>> > > > > jonzeolla/bro-pkg.index> > >>>> > > > > > > and > >>>> > > > > > > >> get it working with a fresh bro 2.5 install when > >>>> attending > >>>> > > the > >>>> > > > > brocon > >>>> > > > > > > talk > >>>> > > > > > > >> < > >>>> > https://www.bro.org/brocon2016/brocon2016_abstracts.html# > >>>> > > > > > > >> bro-packagemanager> > >>>> > > > > > > >> (recording <https://www.youtube.com/watch > >>>> ?v=9RFfPJeGkcE>, > >>>> > > > slides > >>>> > > > > > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>) > >>>> > that > >>>> > > > > introduced > >>>> > > > > > > >> this > >>>> > > > > > > >> to me in the first place. > >>>> > > > > > > >> > >>>> > > > > > > >> Jon > >>>> > > > > > > >> -- > >>>> > > > > > > >> > >>>> > > > > > > >> Jon > >>>> > > > > > > >> > >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > -- > >>>> > > > > > > >>>> > > > > > Jon > >>>> > > > > > > >>>> > > > > > >>>> > > > > -- > >>>> > > > > > >>>> > > > > Jon > >>>> > > > > > >>>> > > > > >>>> > > > -- > >>>> > > > > >>>> > > > Jon > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > -- > >>>> > > > > >>>> > > > Jon > >>>> > > > > >>>> > > > -- > >>>> > > > > >>>> > > > Jon > >>>> > > > > >>>> > > -- > >>>> > > > >>>> > > Jon > >>>> > > > >>>> > > >>>> -- > >>>> > >>>> Jon > > ------------------- > Thank you, > > James Sirota > PPMC- Apache Metron (Incubating) > jsirota AT apache DOT org >
