I don't mind incubator-metron-bro-plugin-kafka 05.04.2017, 13:15, "Nick Allen" <[email protected]>: > Ok, I found that CouchDB and Nifi, among others, use multiple repos. A > fairly obvious pattern seems to emerge. :) > > git://git.apache.org/couchdb-fabric.git > git://git.apache.org/couchdb-couch-replicator.git > git://git.apache.org/couchdb-chttpd.git > > git://git.apache.org/nifi > git://git.apache.org/nifi-site.git > > > > So what shall we call the new repo? Using > "incubator-metron-bro-plugin-kafka" is the obvious choice, but seems > excessively long. Totally open to better name for the plugin itself. > > On Wed, Apr 5, 2017 at 4:00 PM, Nick Allen <[email protected]> wrote: > >> Does anyone know any other Apache projects that are using multiple repos? >> I'd like to see what they've done just so we don't break convention. >> >> On Wed, Apr 5, 2017 at 3:22 PM, Nick Allen <[email protected]> wrote: >> >>> Yes, I will open an INFRA ticket. Just give me a little time to research >>> what we need. >>> >>> On Wed, Apr 5, 2017 at 1:29 PM, [email protected] <[email protected]> >>> wrote: >>> >>>> Okay great, thanks. Would you mind throwing in an INFRA ticket for the >>>> new >>>> repo? I can take it all from there. >>>> >>>> Does anybody know if we have ASF resources to help answer the above legal >>>> question? >>>> >>>> Jon >>>> >>>> On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <[email protected]> wrote: >>>> >>>> > (1) I am not sure if licensing is a problem here. >>>> > >>>> > (2) I am OK with whatever we need to get this effort done and under >>>> ASF. >>>> > >>>> > >>>> > On Wed, Apr 5, 2017 at 1:12 PM, [email protected] <[email protected]> >>>> wrote: >>>> > >>>> > > I'm working on this >>>> > > <https://github.com/JonZeolla/incubator-metron/tree/METRON-348> in >>>> > > preparation for the new repo and migration to a package. It looks >>>> like >>>> > in >>>> > > bro-plugins COPYING >>>> > > <https://github.com/bro/bro-plugins/blob/master/kafka/COPYING> >>>> > attributes >>>> > > to Nick, but in our version COPYING >>>> > > <https://github.com/apache/incubator-metron/tree/master/ >>>> > > metron-sensors/bro-plugin-kafka/COPYING> >>>> > > points to the Apache License. Same with MAINTAINER (this >>>> > > <https://github.com/apache/incubator-metron/blob/master/ >>>> > > metron-sensors/bro-plugin-kafka/MAINTAINER> >>>> > > vs this <https://github.com/bro/bro-plugins/blob/master/kafka/MAINTA >>>> INER >>>> > > >). >>>> > > I assume when we package this up and host it in Apache we need to >>>> give it >>>> > > the Apache license, and point to Metron for MAINTAINER. My questions >>>> > are: >>>> > > >>>> > > 1. Is there any legal/licensing concern here? I am taking changes >>>> from >>>> > > the bro-plugins version and pulling it into the Apache-hosted code. >>>> > IANAL >>>> > > 2. Nick - are you OK with these changes? >>>> > > >>>> > > Jon >>>> > > >>>> > > On Mon, Apr 3, 2017 at 3:50 PM [email protected] <[email protected]> >>>> > wrote: >>>> > > >>>> > > > Can someone on the PMC submit a ticket to INFRA? It looks like >>>> > > > <https://www.apache.org/dev/infra-contact> committers aren't >>>> supposed >>>> > > to. >>>> > > > >>>> > > > Jon >>>> > > > >>>> > > > On Fri, Mar 31, 2017 at 4:23 PM [email protected] <[email protected] >>>> > >>>> > > wrote: >>>> > > > >>>> > > > I would be happy to try it again but I attempted to do that before >>>> with >>>> > > > bro packages and it failed to be able to handle it. I also tried >>>> using >>>> > > > branches of a repo with bro but that similarly failed (and was a >>>> pretty >>>> > > bad >>>> > > > idea to start with). >>>> > > > >>>> > > > Jon >>>> > > > >>>> > > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <[email protected]> wrote: >>>> > > > >>>> > > > We should be able to request just one alternate repo from INFRA, >>>> and >>>> > put >>>> > > a >>>> > > > top hierarchical level in it that doesn’t include a maven pom. As >>>> far >>>> > as >>>> > > > maven and clients are concerned, it >>>> > > > >>>> > > > just increases by 1 the path length to the root of the repo. >>>> > > > >>>> > > > On 3/31/17, 10:30 AM, "[email protected]" <[email protected]> wrote: >>>> > > > >>>> > > > Once we agree on a repo location to host this, I would be >>>> happy to >>>> > > put >>>> > > > together the package and update our environments to use >>>> bro-pkg to >>>> > > > install >>>> > > > the plugin. I have created METRON-813 >>>> > > > <https://issues.apache.org/jira/browse/METRON-813> to track >>>> this >>>> > and >>>> > > > changed METRON-348 < >>>> > https://issues.apache.org/jira/browse/METRON-348 >>>> > > > >>>> > > > to be >>>> > > > a sub-task. >>>> > > > >>>> > > > Otto - the bro packages model doesn't allow colocation with >>>> > anything >>>> > > > else. >>>> > > > That said, if we have two similar situations, and given the >>>> INFRA >>>> > > > example >>>> > > > <https://issues.apache.org/jira/browse/INFRA-7060> Casey >>>> linked to >>>> > > > before >>>> > > > was requesting 9 repos, perhaps we just request two repos. >>>> Would >>>> > > > someone >>>> > > > else mind putting that request in? >>>> > > > >>>> > > > Jon >>>> > > > >>>> > > > On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler < >>>> > > [email protected]> >>>> > > > wrote: >>>> > > > >>>> > > > Could we create a separate repo for more than on thing? like >>>> put … >>>> > > um >>>> > > > let’s say >>>> > > > a maven plugin and the bro plugin? >>>> > > > >>>> > > > >>>> > > > >>>> > > > On March 31, 2017 at 12:30:25, Nick Allen ([email protected]) >>>> > > wrote: >>>> > > > >>>> > > > I agree with everything that I've read. >>>> > > > >>>> > > > One of the guys from Bro had contacted me a while back, >>>> letting me >>>> > > know >>>> > > > that the packaging mechanism in Bro was ready for public >>>> > > consumption. I >>>> > > > just have not had cycles to do anything with it yet. They are >>>> not >>>> > > > wanting >>>> > > > to host any of the plugins. >>>> > > > >>>> > > > I thought the package mechanism requires that a package live >>>> within >>>> > > > its own >>>> > > > repo (which Casey confirmed). This put me in a bind on how to >>>> > tackle >>>> > > > this. I don't want to personally host the plugin in my own >>>> Github >>>> > > > repo. I >>>> > > > would prefer that we host it in a community repo; either Bro or >>>> > > Metron. >>>> > > > Since Bro is moving away from hosting their own plugins, that >>>> > leaves >>>> > > > Metron. >>>> > > > >>>> > > > It would be great if we could create a separate repo for the >>>> > plugin. >>>> > > > That >>>> > > > solves the challenge of using the packaging mechanism. >>>> > > > >>>> > > > We do need to reconcile what is in bro/bro-plugins and what is >>>> in >>>> > > > Metron. >>>> > > > There are some enhancements that I and others have made that >>>> never >>>> > > > made it >>>> > > > back into Metron. They never made it back, because the original >>>> > plan >>>> > > > was >>>> > > > just to switch to using the plugin from bro/bro-plugins before >>>> the >>>> > > > idea of >>>> > > > a packaging mechanism hit Bro. Reconciling should be fairly >>>> easy to >>>> > > > see by >>>> > > > just doing a diff. >>>> > > > >>>> > > > It would be great if others want to take on any of that work. I >>>> > would >>>> > > > be >>>> > > > glad to offer any support that you need. Thanks, Jon! >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > On Thu, Mar 30, 2017 at 11:20 PM, [email protected] < >>>> > [email protected] >>>> > > > >>>> > > > wrote: >>>> > > > >>>> > > > > Ok, great. >>>> > > > > >>>> > > > > I agree, I definitely want to hear from Nick on the topic. My >>>> > team >>>> > > is >>>> > > > > currently looking into enhancing the plugin as well to >>>> > potentially >>>> > > > allow >>>> > > > > sending to multiple clusters, investigating some issues we >>>> see >>>> > when >>>> > > > our >>>> > > > bro >>>> > > > > cluster is under load, turn it into a package, etc. >>>> > > > > >>>> > > > > The work you just did was on our to do list as well so I'm >>>> very >>>> > > > excited >>>> > > > to >>>> > > > > see it come through. >>>> > > > > >>>> > > > > Jon >>>> > > > > >>>> > > > > On Thu, Mar 30, 2017, 11:16 PM Casey Stella < >>>> [email protected]> >>>> > > > wrote: >>>> > > > > >>>> > > > > I *think* it's possible. People do ask for mirrors of >>>> directories >>>> > > > from >>>> > > > > time to time (see >>>> > https://issues.apache.org/jira/browse/INFRA-7060 >>>> > > ). >>>> > > > If >>>> > > > > we >>>> > > > > think this is a good idea, we can pose it to INFRA as a >>>> request. >>>> > > I'd >>>> > > > love >>>> > > > > to see us be able to use the bro packaging infrastructure >>>> and get >>>> > > > more >>>> > > > > visibility for the plugin. >>>> > > > > >>>> > > > > I'd be particularly interested in Nick's opinion on this, >>>> though. >>>> > > > > >>>> > > > > On Thu, Mar 30, 2017 at 11:12 PM, [email protected] < >>>> > > [email protected] >>>> > > > > >>>> > > > > wrote: >>>> > > > > >>>> > > > > > You can version packages - >>>> > > > > > http://bro-package-manager.readthedocs.io/en/stable/ >>>> > > > > package.html#package- >>>> > > > > > versioning >>>> > > > > > >>>> > > > > > I agree that having a separate repo provided by Apache >>>> would be >>>> > > > optimal, >>>> > > > > I >>>> > > > > > just don't know the process for that or if it was even >>>> > reasonable >>>> > > > to >>>> > > > > > suggest. >>>> > > > > > >>>> > > > > > Jon >>>> > > > > > >>>> > > > > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella < >>>> > [email protected]> >>>> > > > wrote: >>>> > > > > > >>>> > > > > > > Looking at the bro packages, it appears that bro is >>>> expecting >>>> > > > things >>>> > > > to >>>> > > > > > be >>>> > > > > > > its own git repository. I wonder if we could either >>>> request >>>> > > INFRA >>>> > > > > > provide >>>> > > > > > > another repo for the bro-kafka plugin and integrate it >>>> into >>>> > > > metron as >>>> > > > a >>>> > > > > > git >>>> > > > > > > submodule *or* if we could request INFRA to create a >>>> github >>>> > > > mirror of >>>> > > > > the >>>> > > > > > > metron-sensors/bro-kafka-plugin directory. I'm not sure >>>> how >>>> > > > viable >>>> > > > > > either >>>> > > > > > > of those options are, frankly. >>>> > > > > > > >>>> > > > > > > One thing that I didn't see is how do you specify a >>>> > particular >>>> > > > release >>>> > > > > of >>>> > > > > > > the plugin that you want to install? For us, we'd want to >>>> > > > release the >>>> > > > > > > plugin along with the product. I didn't quite see how >>>> you'd >>>> > > push >>>> > > > > > releases >>>> > > > > > > for bro plugins. >>>> > > > > > > >>>> > > > > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella < >>>> > > > [email protected]> >>>> > > > > > wrote: >>>> > > > > > > >>>> > > > > > > > So, I do agree with the concern. Is there a way to >>>> host the >>>> > > > package >>>> > > > > > > > within Metron? I definitely would like to see the >>>> > > > modifications at >>>> > > > > > > > https://github.com/bro/bro-plu >>>> gins/commit/b9f1f35415cb0db >>>> > > > > > > > 065348da0a5043a8353b4a0a8 brought back into Metron and >>>> I'd >>>> > > > love for >>>> > > > > us >>>> > > > > > to >>>> > > > > > > > host the plugin. >>>> > > > > > > > >>>> > > > > > > > Thoughts? >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > On Thu, Mar 30, 2017 at 9:09 PM, [email protected] < >>>> > > > [email protected]> >>>> > > > >>>> > > > > > > > wrote: >>>> > > > > > > > >>>> > > > > > > >> Today I was taking a look at METRON-812 >>>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-812>, >>>> which >>>> > > > made me >>>> > > > > > > recall >>>> > > > > > > >> some conversations from a while back regarding where >>>> the >>>> > bro >>>> > > > kafka >>>> > > > > > > plugin >>>> > > > > > > >> should ultimately live, and how to update it. >>>> > > > > > > >> >>>> > > > > > > >> Back in METRON-348 <https://issues.apache.org/ >>>> > > > > jira/browse/METRON-348> >>>> > > > > > I >>>> > > > > > > >> brought up the fact that some important changes >>>> > > > > > > >> <https://github.com/bro/bro-plugins/commit/ >>>> > > b9f1f35415cb0db06 >>>> > > > > > > >> 5348da0a5043a8353b4a0a8> >>>> > > > > > > >> were made to the externally hosted version of the >>>> kafka >>>> > > > plugin, >>>> > > > and >>>> > > > > > were >>>> > > > > > > >> never introduced to Metron's hosted version (i.e. the >>>> one >>>> > we >>>> > > > use >>>> > > > > > > >> <https://github.com/apache/inc >>>> ubator-metron/blob/master/ >>>> > > metr >>>> > > > > > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml> >>>> > > > > > > >> in vagrant when bro is installed). The conversation >>>> went >>>> > > down >>>> > > > the >>>> > > > > > route >>>> > > > > > > >> of >>>> > > > > > > >> discussing whether or not the bro kafka plugin code >>>> should >>>> > > > continue >>>> > > > > to >>>> > > > > > > >> live >>>> > > > > > > >> in Metron in the first place. Now, with METRON-812, I >>>> see >>>> > us >>>> > > > > further >>>> > > > > > > >> muddying the waters of where to go for the right >>>> plugin, >>>> > as >>>> > > > our >>>> > > > > > version >>>> > > > > > > is >>>> > > > > > > >> still missing the public changes but adds some very >>>> > > important >>>> > > > new >>>> > > > > > > >> functionality. >>>> > > > > > > >> >>>> > > > > > > >> I'd like to bring up the idea of using bro's packages >>>> > > > > > > >> <https://github.com/bro/packages> framework, >>>> released in >>>> > > > late 2016 >>>> > > > > > > >> < >>>> > > > http://blog.bro.org/2016/10/introducing-bro-package-manager.html> >>>> > > > > > > >> (additional >>>> > > > > > > >> documentation here < >>>> > > > > > > http://bro-package-manager.readthedocs.io/en/stable/ >>>> > > > > > > >> >), >>>> > > > > > > >> as a potential place for this to be hosted/referenced. >>>> > This >>>> > > > is a >>>> > > > > > simple >>>> > > > > > > >> and supported method (funded by Mozilla >>>> > > > > > > >> <https://blog.mozilla.org/blog >>>> /2015/12/10/mozilla-open- >>>> > > sourc >>>> > > > > > > >> e-support-first-awards-made/>) >>>> > > > > > > >> to install and uninstall bro scripts, plugins, etc., >>>> and >>>> > it >>>> > > > also >>>> > > > > > allows >>>> > > > > > > us >>>> > > > > > > >> to continue to have enough control over updates to the >>>> > > plugin >>>> > > > so >>>> > > > > that >>>> > > > > > it >>>> > > > > > > >> will not slow down Metron development by having it as >>>> a >>>> > > > dependency >>>> > > > > > > >> (resolving both of Casey's concerns noted here >>>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348? >>>> > > focusedCom >>>> > > > > > > >> mentId=15391865&page=com.atlas >>>> sian.jira.plugin.system. >>>> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391865>, >>>> > > > > > > >> and I think this solution is supported by Nick's >>>> comments >>>> > > here >>>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348? >>>> > > focusedCom >>>> > > > > > > >> mentId=15391872&page=com.atlas >>>> sian.jira.plugin.system. >>>> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391872> >>>> > > > > > > >> as >>>> > > > > > > >> well). >>>> > > > > > > >> >>>> > > > > > > >> The only thing I'm not sure about is where to host the >>>> > > plugin >>>> > > > itself >>>> > > > > - >>>> > > > > > > my >>>> > > > > > > >> first thought would be Nick's github < >>>> > > > https://github.com/nickwallen >>>> > > > > >, >>>> > > > > > > as >>>> > > > > > > >> he >>>> > > > > > > >> really kicked off this effort, but maybe we can think >>>> of >>>> > > > something >>>> > > > > > > better. >>>> > > > > > > >> >>>> > > > > > > >> Is this approach of interest to anybody? It is >>>> extremely >>>> > > > simple to >>>> > > > > > put >>>> > > > > > > >> together - I was able to throw one together >>>> > > > > > > >> <https://github.com/bro/packages/blob/master/ >>>> > > > > jonzeolla/bro-pkg.index> >>>> > > > > > > and >>>> > > > > > > >> get it working with a fresh bro 2.5 install when >>>> attending >>>> > > the >>>> > > > > brocon >>>> > > > > > > talk >>>> > > > > > > >> < >>>> > https://www.bro.org/brocon2016/brocon2016_abstracts.html# >>>> > > > > > > >> bro-packagemanager> >>>> > > > > > > >> (recording <https://www.youtube.com/watch >>>> ?v=9RFfPJeGkcE>, >>>> > > > slides >>>> > > > > > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>) >>>> > that >>>> > > > > introduced >>>> > > > > > > >> this >>>> > > > > > > >> to me in the first place. >>>> > > > > > > >> >>>> > > > > > > >> Jon >>>> > > > > > > >> -- >>>> > > > > > > >> >>>> > > > > > > >> Jon >>>> > > > > > > >> >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > -- >>>> > > > > > >>>> > > > > > Jon >>>> > > > > > >>>> > > > > >>>> > > > > -- >>>> > > > > >>>> > > > > Jon >>>> > > > > >>>> > > > >>>> > > > -- >>>> > > > >>>> > > > Jon >>>> > > > >>>> > > > >>>> > > > >>>> > > > -- >>>> > > > >>>> > > > Jon >>>> > > > >>>> > > > -- >>>> > > > >>>> > > > Jon >>>> > > > >>>> > > -- >>>> > > >>>> > > Jon >>>> > > >>>> > >>>> -- >>>> >>>> Jon
------------------- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
