Ok, I found that CouchDB and Nifi, among others, use multiple repos. A fairly obvious pattern seems to emerge. :)
git://git.apache.org/couchdb-fabric.git git://git.apache.org/couchdb-couch-replicator.git git://git.apache.org/couchdb-chttpd.git git://git.apache.org/nifi git://git.apache.org/nifi-site.git So what shall we call the new repo? Using "incubator-metron-bro-plugin-kafka" is the obvious choice, but seems excessively long. Totally open to better name for the plugin itself. On Wed, Apr 5, 2017 at 4:00 PM, Nick Allen <n...@nickallen.org> wrote: > Does anyone know any other Apache projects that are using multiple repos? > I'd like to see what they've done just so we don't break convention. > > > > > > > On Wed, Apr 5, 2017 at 3:22 PM, Nick Allen <n...@nickallen.org> wrote: > >> Yes, I will open an INFRA ticket. Just give me a little time to research >> what we need. >> >> On Wed, Apr 5, 2017 at 1:29 PM, zeo...@gmail.com <zeo...@gmail.com> >> wrote: >> >>> Okay great, thanks. Would you mind throwing in an INFRA ticket for the >>> new >>> repo? I can take it all from there. >>> >>> Does anybody know if we have ASF resources to help answer the above legal >>> question? >>> >>> Jon >>> >>> On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <n...@nickallen.org> wrote: >>> >>> > (1) I am not sure if licensing is a problem here. >>> > >>> > (2) I am OK with whatever we need to get this effort done and under >>> ASF. >>> > >>> > >>> > On Wed, Apr 5, 2017 at 1:12 PM, zeo...@gmail.com <zeo...@gmail.com> >>> wrote: >>> > >>> > > I'm working on this >>> > > <https://github.com/JonZeolla/incubator-metron/tree/METRON-348> in >>> > > preparation for the new repo and migration to a package. It looks >>> like >>> > in >>> > > bro-plugins COPYING >>> > > <https://github.com/bro/bro-plugins/blob/master/kafka/COPYING> >>> > attributes >>> > > to Nick, but in our version COPYING >>> > > <https://github.com/apache/incubator-metron/tree/master/ >>> > > metron-sensors/bro-plugin-kafka/COPYING> >>> > > points to the Apache License. Same with MAINTAINER (this >>> > > <https://github.com/apache/incubator-metron/blob/master/ >>> > > metron-sensors/bro-plugin-kafka/MAINTAINER> >>> > > vs this <https://github.com/bro/bro-plugins/blob/master/kafka/MAINTA >>> INER >>> > > >). >>> > > I assume when we package this up and host it in Apache we need to >>> give it >>> > > the Apache license, and point to Metron for MAINTAINER. My questions >>> > are: >>> > > >>> > > 1. Is there any legal/licensing concern here? I am taking changes >>> from >>> > > the bro-plugins version and pulling it into the Apache-hosted code. >>> > IANAL >>> > > 2. Nick - are you OK with these changes? >>> > > >>> > > Jon >>> > > >>> > > On Mon, Apr 3, 2017 at 3:50 PM zeo...@gmail.com <zeo...@gmail.com> >>> > wrote: >>> > > >>> > > > Can someone on the PMC submit a ticket to INFRA? It looks like >>> > > > <https://www.apache.org/dev/infra-contact> committers aren't >>> supposed >>> > > to. >>> > > > >>> > > > Jon >>> > > > >>> > > > On Fri, Mar 31, 2017 at 4:23 PM zeo...@gmail.com <zeo...@gmail.com >>> > >>> > > wrote: >>> > > > >>> > > > I would be happy to try it again but I attempted to do that before >>> with >>> > > > bro packages and it failed to be able to handle it. I also tried >>> using >>> > > > branches of a repo with bro but that similarly failed (and was a >>> pretty >>> > > bad >>> > > > idea to start with). >>> > > > >>> > > > Jon >>> > > > >>> > > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <ma...@apache.org> wrote: >>> > > > >>> > > > We should be able to request just one alternate repo from INFRA, >>> and >>> > put >>> > > a >>> > > > top hierarchical level in it that doesn’t include a maven pom. As >>> far >>> > as >>> > > > maven and clients are concerned, it >>> > > > >>> > > > just increases by 1 the path length to the root of the repo. >>> > > > >>> > > > On 3/31/17, 10:30 AM, "zeo...@gmail.com" <zeo...@gmail.com> wrote: >>> > > > >>> > > > Once we agree on a repo location to host this, I would be >>> happy to >>> > > put >>> > > > together the package and update our environments to use >>> bro-pkg to >>> > > > install >>> > > > the plugin. I have created METRON-813 >>> > > > <https://issues.apache.org/jira/browse/METRON-813> to track >>> this >>> > and >>> > > > changed METRON-348 < >>> > https://issues.apache.org/jira/browse/METRON-348 >>> > > > >>> > > > to be >>> > > > a sub-task. >>> > > > >>> > > > Otto - the bro packages model doesn't allow colocation with >>> > anything >>> > > > else. >>> > > > That said, if we have two similar situations, and given the >>> INFRA >>> > > > example >>> > > > <https://issues.apache.org/jira/browse/INFRA-7060> Casey >>> linked to >>> > > > before >>> > > > was requesting 9 repos, perhaps we just request two repos. >>> Would >>> > > > someone >>> > > > else mind putting that request in? >>> > > > >>> > > > Jon >>> > > > >>> > > > On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler < >>> > > ottobackwa...@gmail.com> >>> > > > wrote: >>> > > > >>> > > > Could we create a separate repo for more than on thing? like >>> put … >>> > > um >>> > > > let’s say >>> > > > a maven plugin and the bro plugin? >>> > > > >>> > > > >>> > > > >>> > > > On March 31, 2017 at 12:30:25, Nick Allen (n...@nickallen.org) >>> > > wrote: >>> > > > >>> > > > I agree with everything that I've read. >>> > > > >>> > > > One of the guys from Bro had contacted me a while back, >>> letting me >>> > > know >>> > > > that the packaging mechanism in Bro was ready for public >>> > > consumption. I >>> > > > just have not had cycles to do anything with it yet. They are >>> not >>> > > > wanting >>> > > > to host any of the plugins. >>> > > > >>> > > > I thought the package mechanism requires that a package live >>> within >>> > > > its own >>> > > > repo (which Casey confirmed). This put me in a bind on how to >>> > tackle >>> > > > this. I don't want to personally host the plugin in my own >>> Github >>> > > > repo. I >>> > > > would prefer that we host it in a community repo; either Bro or >>> > > Metron. >>> > > > Since Bro is moving away from hosting their own plugins, that >>> > leaves >>> > > > Metron. >>> > > > >>> > > > It would be great if we could create a separate repo for the >>> > plugin. >>> > > > That >>> > > > solves the challenge of using the packaging mechanism. >>> > > > >>> > > > We do need to reconcile what is in bro/bro-plugins and what is >>> in >>> > > > Metron. >>> > > > There are some enhancements that I and others have made that >>> never >>> > > > made it >>> > > > back into Metron. They never made it back, because the original >>> > plan >>> > > > was >>> > > > just to switch to using the plugin from bro/bro-plugins before >>> the >>> > > > idea of >>> > > > a packaging mechanism hit Bro. Reconciling should be fairly >>> easy to >>> > > > see by >>> > > > just doing a diff. >>> > > > >>> > > > It would be great if others want to take on any of that work. I >>> > would >>> > > > be >>> > > > glad to offer any support that you need. Thanks, Jon! >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > On Thu, Mar 30, 2017 at 11:20 PM, zeo...@gmail.com < >>> > zeo...@gmail.com >>> > > > >>> > > > wrote: >>> > > > >>> > > > > Ok, great. >>> > > > > >>> > > > > I agree, I definitely want to hear from Nick on the topic. My >>> > team >>> > > is >>> > > > > currently looking into enhancing the plugin as well to >>> > potentially >>> > > > allow >>> > > > > sending to multiple clusters, investigating some issues we >>> see >>> > when >>> > > > our >>> > > > bro >>> > > > > cluster is under load, turn it into a package, etc. >>> > > > > >>> > > > > The work you just did was on our to do list as well so I'm >>> very >>> > > > excited >>> > > > to >>> > > > > see it come through. >>> > > > > >>> > > > > Jon >>> > > > > >>> > > > > On Thu, Mar 30, 2017, 11:16 PM Casey Stella < >>> ceste...@gmail.com> >>> > > > wrote: >>> > > > > >>> > > > > I *think* it's possible. People do ask for mirrors of >>> directories >>> > > > from >>> > > > > time to time (see >>> > https://issues.apache.org/jira/browse/INFRA-7060 >>> > > ). >>> > > > If >>> > > > > we >>> > > > > think this is a good idea, we can pose it to INFRA as a >>> request. >>> > > I'd >>> > > > love >>> > > > > to see us be able to use the bro packaging infrastructure >>> and get >>> > > > more >>> > > > > visibility for the plugin. >>> > > > > >>> > > > > I'd be particularly interested in Nick's opinion on this, >>> though. >>> > > > > >>> > > > > On Thu, Mar 30, 2017 at 11:12 PM, zeo...@gmail.com < >>> > > zeo...@gmail.com >>> > > > > >>> > > > > wrote: >>> > > > > >>> > > > > > You can version packages - >>> > > > > > http://bro-package-manager.readthedocs.io/en/stable/ >>> > > > > package.html#package- >>> > > > > > versioning >>> > > > > > >>> > > > > > I agree that having a separate repo provided by Apache >>> would be >>> > > > optimal, >>> > > > > I >>> > > > > > just don't know the process for that or if it was even >>> > reasonable >>> > > > to >>> > > > > > suggest. >>> > > > > > >>> > > > > > Jon >>> > > > > > >>> > > > > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella < >>> > ceste...@gmail.com> >>> > > > wrote: >>> > > > > > >>> > > > > > > Looking at the bro packages, it appears that bro is >>> expecting >>> > > > things >>> > > > to >>> > > > > > be >>> > > > > > > its own git repository. I wonder if we could either >>> request >>> > > INFRA >>> > > > > > provide >>> > > > > > > another repo for the bro-kafka plugin and integrate it >>> into >>> > > > metron as >>> > > > a >>> > > > > > git >>> > > > > > > submodule *or* if we could request INFRA to create a >>> github >>> > > > mirror of >>> > > > > the >>> > > > > > > metron-sensors/bro-kafka-plugin directory. I'm not sure >>> how >>> > > > viable >>> > > > > > either >>> > > > > > > of those options are, frankly. >>> > > > > > > >>> > > > > > > One thing that I didn't see is how do you specify a >>> > particular >>> > > > release >>> > > > > of >>> > > > > > > the plugin that you want to install? For us, we'd want to >>> > > > release the >>> > > > > > > plugin along with the product. I didn't quite see how >>> you'd >>> > > push >>> > > > > > releases >>> > > > > > > for bro plugins. >>> > > > > > > >>> > > > > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella < >>> > > > ceste...@gmail.com> >>> > > > > > wrote: >>> > > > > > > >>> > > > > > > > So, I do agree with the concern. Is there a way to >>> host the >>> > > > package >>> > > > > > > > within Metron? I definitely would like to see the >>> > > > modifications at >>> > > > > > > > https://github.com/bro/bro-plu >>> gins/commit/b9f1f35415cb0db >>> > > > > > > > 065348da0a5043a8353b4a0a8 brought back into Metron and >>> I'd >>> > > > love for >>> > > > > us >>> > > > > > to >>> > > > > > > > host the plugin. >>> > > > > > > > >>> > > > > > > > Thoughts? >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > On Thu, Mar 30, 2017 at 9:09 PM, zeo...@gmail.com < >>> > > > zeo...@gmail.com> >>> > > > >>> > > > > > > > wrote: >>> > > > > > > > >>> > > > > > > >> Today I was taking a look at METRON-812 >>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-812>, >>> which >>> > > > made me >>> > > > > > > recall >>> > > > > > > >> some conversations from a while back regarding where >>> the >>> > bro >>> > > > kafka >>> > > > > > > plugin >>> > > > > > > >> should ultimately live, and how to update it. >>> > > > > > > >> >>> > > > > > > >> Back in METRON-348 <https://issues.apache.org/ >>> > > > > jira/browse/METRON-348> >>> > > > > > I >>> > > > > > > >> brought up the fact that some important changes >>> > > > > > > >> <https://github.com/bro/bro-plugins/commit/ >>> > > b9f1f35415cb0db06 >>> > > > > > > >> 5348da0a5043a8353b4a0a8> >>> > > > > > > >> were made to the externally hosted version of the >>> kafka >>> > > > plugin, >>> > > > and >>> > > > > > were >>> > > > > > > >> never introduced to Metron's hosted version (i.e. the >>> one >>> > we >>> > > > use >>> > > > > > > >> <https://github.com/apache/inc >>> ubator-metron/blob/master/ >>> > > metr >>> > > > > > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml> >>> > > > > > > >> in vagrant when bro is installed). The conversation >>> went >>> > > down >>> > > > the >>> > > > > > route >>> > > > > > > >> of >>> > > > > > > >> discussing whether or not the bro kafka plugin code >>> should >>> > > > continue >>> > > > > to >>> > > > > > > >> live >>> > > > > > > >> in Metron in the first place. Now, with METRON-812, I >>> see >>> > us >>> > > > > further >>> > > > > > > >> muddying the waters of where to go for the right >>> plugin, >>> > as >>> > > > our >>> > > > > > version >>> > > > > > > is >>> > > > > > > >> still missing the public changes but adds some very >>> > > important >>> > > > new >>> > > > > > > >> functionality. >>> > > > > > > >> >>> > > > > > > >> I'd like to bring up the idea of using bro's packages >>> > > > > > > >> <https://github.com/bro/packages> framework, >>> released in >>> > > > late 2016 >>> > > > > > > >> < >>> > > > http://blog.bro.org/2016/10/introducing-bro-package-manager.html> >>> > > > > > > >> (additional >>> > > > > > > >> documentation here < >>> > > > > > > http://bro-package-manager.readthedocs.io/en/stable/ >>> > > > > > > >> >), >>> > > > > > > >> as a potential place for this to be hosted/referenced. >>> > This >>> > > > is a >>> > > > > > simple >>> > > > > > > >> and supported method (funded by Mozilla >>> > > > > > > >> <https://blog.mozilla.org/blog >>> /2015/12/10/mozilla-open- >>> > > sourc >>> > > > > > > >> e-support-first-awards-made/>) >>> > > > > > > >> to install and uninstall bro scripts, plugins, etc., >>> and >>> > it >>> > > > also >>> > > > > > allows >>> > > > > > > us >>> > > > > > > >> to continue to have enough control over updates to the >>> > > plugin >>> > > > so >>> > > > > that >>> > > > > > it >>> > > > > > > >> will not slow down Metron development by having it as >>> a >>> > > > dependency >>> > > > > > > >> (resolving both of Casey's concerns noted here >>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348? >>> > > focusedCom >>> > > > > > > >> mentId=15391865&page=com.atlas >>> sian.jira.plugin.system. >>> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391865>, >>> > > > > > > >> and I think this solution is supported by Nick's >>> comments >>> > > here >>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348? >>> > > focusedCom >>> > > > > > > >> mentId=15391872&page=com.atlas >>> sian.jira.plugin.system. >>> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391872> >>> > > > > > > >> as >>> > > > > > > >> well). >>> > > > > > > >> >>> > > > > > > >> The only thing I'm not sure about is where to host the >>> > > plugin >>> > > > itself >>> > > > > - >>> > > > > > > my >>> > > > > > > >> first thought would be Nick's github < >>> > > > https://github.com/nickwallen >>> > > > > >, >>> > > > > > > as >>> > > > > > > >> he >>> > > > > > > >> really kicked off this effort, but maybe we can think >>> of >>> > > > something >>> > > > > > > better. >>> > > > > > > >> >>> > > > > > > >> Is this approach of interest to anybody? It is >>> extremely >>> > > > simple to >>> > > > > > put >>> > > > > > > >> together - I was able to throw one together >>> > > > > > > >> <https://github.com/bro/packages/blob/master/ >>> > > > > jonzeolla/bro-pkg.index> >>> > > > > > > and >>> > > > > > > >> get it working with a fresh bro 2.5 install when >>> attending >>> > > the >>> > > > > brocon >>> > > > > > > talk >>> > > > > > > >> < >>> > https://www.bro.org/brocon2016/brocon2016_abstracts.html# >>> > > > > > > >> bro-packagemanager> >>> > > > > > > >> (recording <https://www.youtube.com/watch >>> ?v=9RFfPJeGkcE>, >>> > > > slides >>> > > > > > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>) >>> > that >>> > > > > introduced >>> > > > > > > >> this >>> > > > > > > >> to me in the first place. >>> > > > > > > >> >>> > > > > > > >> Jon >>> > > > > > > >> -- >>> > > > > > > >> >>> > > > > > > >> Jon >>> > > > > > > >> >>> > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > -- >>> > > > > > >>> > > > > > Jon >>> > > > > > >>> > > > > >>> > > > > -- >>> > > > > >>> > > > > Jon >>> > > > > >>> > > > >>> > > > -- >>> > > > >>> > > > Jon >>> > > > >>> > > > >>> > > > >>> > > > -- >>> > > > >>> > > > Jon >>> > > > >>> > > > -- >>> > > > >>> > > > Jon >>> > > > >>> > > -- >>> > > >>> > > Jon >>> > > >>> > >>> -- >>> >>> Jon >>> >> >> >
