Re: [DISCUSS] Metron Release - 0.7.1 next steps
Sure, I can kick it off. I'd expect that to drop either tonight or tomorrow morning, depending on when I can dedicate a bit of time. Thanks to everyone who's helped (and continued to help) with working on this! On Wed, May 8, 2019 at 12:23 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Hey everyone, > > METRON-2100 has been merged - https://github.com/apache/metron/pull/1398, > and the parser aggregation UI work and feature branch is under way. > > Justin, can we kick off an RC2? > > Cheers, > Mike > > On Fri, May 3, 2019 at 6:06 AM Otto Fowler > wrote: > > > Despite the name, we *have* been using it as both for quite some amount > of > > time. It *is* both dev and demo, and we recommend it as such on the list > > all the time. > > > > So there isn’t a decision to be made here as far as the status quo -> we > > use full dev as both dev and demo. > > > > > > > > > > On May 2, 2019 at 18:53:37, Michael Miklavcic ( > michael.miklav...@gmail.com > > ) > > wrote: > > > > Whether or not full dev is, first and foremost, "dev" I think your > > questions being up a good point. If not full_dev for introducing new > users, > > then what? If we want to provide a different env for letting people > tinker > > and try it out than we do for development, that's completely fine. But we > > don't have that right now. So we can treat full_dev as multipurpose, or > we > > can stop directing non-devs to it, or we can add something new. I > honestly > > don't have any recommendations here. We've talked about docker instances > > for replacing in-memory components, but I'm still not sure that solves > this > > problem, or adds more complexity. Given the current options on the table, > > I'm inclined to go with "full_dev" serves both dev and demo purposes. > Otto, > > what do you think? > > > > On Thu, May 2, 2019, 4:32 PM Otto Fowler > wrote: > > > > > I’ve commented on the PR, and I won’t repeat it here as well, I will > > > however ask again if we know and can list all of the usability issues > > that > > > surround this problem. IE. All the things that can happen or may happen > > > for people who are not Metron developers and committers who are using > > > full dev, because we keep recommending it. > > > > > > > > > > > > On May 2, 2019 at 17:38:30, Michael Miklavcic ( > > michael.miklav...@gmail.com > > > ) > > > wrote: > > > > > > PR is up. I added the doc change to the metron-deployment README since > > this > > > serves as the gateway doc for all the VM instances. All of which would > be > > > affected by the feature gap. > > > > > > https://github.com/apache/metron/pull/1398 > > > > > > On Thu, May 2, 2019 at 1:37 PM Michael Miklavcic < > > > michael.miklav...@gmail.com> wrote: > > > > > > > Here's the ticket I created to track it, which also references the > Jira > > > > for the new UI feature. > > > > https://issues.apache.org/jira/browse/METRON-2100 > > > > > > > > On Thu, May 2, 2019 at 12:34 PM Michael Miklavcic < > > > > michael.miklav...@gmail.com> wrote: > > > > > > > >> :-) > > > >> > > > >> I expect to have #2 out sometime today. > > > >> > > > >> On Thu, May 2, 2019, 12:11 PM Justin Leet > > > wrote: > > > >> > > > >>> > > > > >>> > I personally > > > >>> > don't like this feature gap in full dev. It seems Otto agrees, > and > > > >>> Casey at > > > >>> > the very least sees it as enough of an issue to gate us from 0.8. > > > >>> > > > > >>> > > > >>> +1 on all of this. I don't like it either. > > > >>> > > > >>> > > > >>> > Our vote landed 2-2. We are having a discussion about what to do > > with > > > >>> the > > > >>> > release. This is that discussion. > > > >>> > > > >>> > > > >>> I'm going to be honest, my response was a combination of misreading > > > what > > > >>> you said and thinking you were proposing delaying the release more > > > >>> seriously and feeling a bit blindsided by a perceived move from the > > > >>> initial > > > >>> "take more time than originally anticipated" (which in my head I > took > > > as > > > >>> a > > > >>> couple days) to "versus next week, or the week after" (where > delaying > > > >>> things weeks is something I personally would like not buried so far > > > down > > > >>> in > > > >>> the thread). Totally my bad, sorry about that. > > > >>> > > > >>> Other than that, it sounds like we're pretty much in agreement. > > > >>> > > > >>> Here's my current understanding of the state and consensus as of > > right > > > >>> now > > > >>> (which is subject to change as more discussion happens): > > > >>> > > > >>> - Most of the people in the thread are in favor of #2 for 0.7.1 and > > #3 > > > >>> for 0.8.0. > > > >>> - I don't believe I've seen an explicit response from Otto on what > > > >>> he > > > >>> thinks about doing this, and from a personal perspective like to > > > >>> see what > > > >>> his opinion is as the person who originally brought it up. > > > >>> - Mike said he's going to kick out a PR that addresses #2 > > > >>> - After that undergoes
Re: [DISCUSS] Metron Release - 0.7.1 next steps
Hey everyone, METRON-2100 has been merged - https://github.com/apache/metron/pull/1398, and the parser aggregation UI work and feature branch is under way. Justin, can we kick off an RC2? Cheers, Mike On Fri, May 3, 2019 at 6:06 AM Otto Fowler wrote: > Despite the name, we *have* been using it as both for quite some amount of > time. It *is* both dev and demo, and we recommend it as such on the list > all the time. > > So there isn’t a decision to be made here as far as the status quo -> we > use full dev as both dev and demo. > > > > > On May 2, 2019 at 18:53:37, Michael Miklavcic (michael.miklav...@gmail.com > ) > wrote: > > Whether or not full dev is, first and foremost, "dev" I think your > questions being up a good point. If not full_dev for introducing new users, > then what? If we want to provide a different env for letting people tinker > and try it out than we do for development, that's completely fine. But we > don't have that right now. So we can treat full_dev as multipurpose, or we > can stop directing non-devs to it, or we can add something new. I honestly > don't have any recommendations here. We've talked about docker instances > for replacing in-memory components, but I'm still not sure that solves this > problem, or adds more complexity. Given the current options on the table, > I'm inclined to go with "full_dev" serves both dev and demo purposes. Otto, > what do you think? > > On Thu, May 2, 2019, 4:32 PM Otto Fowler wrote: > > > I’ve commented on the PR, and I won’t repeat it here as well, I will > > however ask again if we know and can list all of the usability issues > that > > surround this problem. IE. All the things that can happen or may happen > > for people who are not Metron developers and committers who are using > > full dev, because we keep recommending it. > > > > > > > > On May 2, 2019 at 17:38:30, Michael Miklavcic ( > michael.miklav...@gmail.com > > ) > > wrote: > > > > PR is up. I added the doc change to the metron-deployment README since > this > > serves as the gateway doc for all the VM instances. All of which would be > > affected by the feature gap. > > > > https://github.com/apache/metron/pull/1398 > > > > On Thu, May 2, 2019 at 1:37 PM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > > > Here's the ticket I created to track it, which also references the Jira > > > for the new UI feature. > > > https://issues.apache.org/jira/browse/METRON-2100 > > > > > > On Thu, May 2, 2019 at 12:34 PM Michael Miklavcic < > > > michael.miklav...@gmail.com> wrote: > > > > > >> :-) > > >> > > >> I expect to have #2 out sometime today. > > >> > > >> On Thu, May 2, 2019, 12:11 PM Justin Leet > > wrote: > > >> > > >>> > > > >>> > I personally > > >>> > don't like this feature gap in full dev. It seems Otto agrees, and > > >>> Casey at > > >>> > the very least sees it as enough of an issue to gate us from 0.8. > > >>> > > > >>> > > >>> +1 on all of this. I don't like it either. > > >>> > > >>> > > >>> > Our vote landed 2-2. We are having a discussion about what to do > with > > >>> the > > >>> > release. This is that discussion. > > >>> > > >>> > > >>> I'm going to be honest, my response was a combination of misreading > > what > > >>> you said and thinking you were proposing delaying the release more > > >>> seriously and feeling a bit blindsided by a perceived move from the > > >>> initial > > >>> "take more time than originally anticipated" (which in my head I took > > as > > >>> a > > >>> couple days) to "versus next week, or the week after" (where delaying > > >>> things weeks is something I personally would like not buried so far > > down > > >>> in > > >>> the thread). Totally my bad, sorry about that. > > >>> > > >>> Other than that, it sounds like we're pretty much in agreement. > > >>> > > >>> Here's my current understanding of the state and consensus as of > right > > >>> now > > >>> (which is subject to change as more discussion happens): > > >>> > > >>> - Most of the people in the thread are in favor of #2 for 0.7.1 and > #3 > > >>> for 0.8.0. > > >>> - I don't believe I've seen an explicit response from Otto on what > > >>> he > > >>> thinks about doing this, and from a personal perspective like to > > >>> see what > > >>> his opinion is as the person who originally brought it up. > > >>> - Mike said he's going to kick out a PR that addresses #2 > > >>> - After that undergoes the normal review process and is merged, we > > >>> proceed normally and cut RC2. > > >>> > > >>> > > >>> On Thu, May 2, 2019 at 1:14 PM Michael Miklavcic < > > >>> michael.miklav...@gmail.com> wrote: > > >>> > > >>> > I think your later point about using a project release version, > from > > >>> the > > >>> > example of using other projects, is a valid one. To exactly that > > >>> point, a > > >>> > community member (Otto) brought up an issue/bug they found through > > >>> testing > > >>> > that they were previously unaware of and did not find documented. > > >>> Which was
Re: [DISCUSS] Metron Release - 0.7.1 next steps
Despite the name, we *have* been using it as both for quite some amount of time. It *is* both dev and demo, and we recommend it as such on the list all the time. So there isn’t a decision to be made here as far as the status quo -> we use full dev as both dev and demo. On May 2, 2019 at 18:53:37, Michael Miklavcic (michael.miklav...@gmail.com) wrote: Whether or not full dev is, first and foremost, "dev" I think your questions being up a good point. If not full_dev for introducing new users, then what? If we want to provide a different env for letting people tinker and try it out than we do for development, that's completely fine. But we don't have that right now. So we can treat full_dev as multipurpose, or we can stop directing non-devs to it, or we can add something new. I honestly don't have any recommendations here. We've talked about docker instances for replacing in-memory components, but I'm still not sure that solves this problem, or adds more complexity. Given the current options on the table, I'm inclined to go with "full_dev" serves both dev and demo purposes. Otto, what do you think? On Thu, May 2, 2019, 4:32 PM Otto Fowler wrote: > I’ve commented on the PR, and I won’t repeat it here as well, I will > however ask again if we know and can list all of the usability issues that > surround this problem. IE. All the things that can happen or may happen > for people who are not Metron developers and committers who are using > full dev, because we keep recommending it. > > > > On May 2, 2019 at 17:38:30, Michael Miklavcic (michael.miklav...@gmail.com > ) > wrote: > > PR is up. I added the doc change to the metron-deployment README since this > serves as the gateway doc for all the VM instances. All of which would be > affected by the feature gap. > > https://github.com/apache/metron/pull/1398 > > On Thu, May 2, 2019 at 1:37 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Here's the ticket I created to track it, which also references the Jira > > for the new UI feature. > > https://issues.apache.org/jira/browse/METRON-2100 > > > > On Thu, May 2, 2019 at 12:34 PM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > >> :-) > >> > >> I expect to have #2 out sometime today. > >> > >> On Thu, May 2, 2019, 12:11 PM Justin Leet > wrote: > >> > >>> > > >>> > I personally > >>> > don't like this feature gap in full dev. It seems Otto agrees, and > >>> Casey at > >>> > the very least sees it as enough of an issue to gate us from 0.8. > >>> > > >>> > >>> +1 on all of this. I don't like it either. > >>> > >>> > >>> > Our vote landed 2-2. We are having a discussion about what to do with > >>> the > >>> > release. This is that discussion. > >>> > >>> > >>> I'm going to be honest, my response was a combination of misreading > what > >>> you said and thinking you were proposing delaying the release more > >>> seriously and feeling a bit blindsided by a perceived move from the > >>> initial > >>> "take more time than originally anticipated" (which in my head I took > as > >>> a > >>> couple days) to "versus next week, or the week after" (where delaying > >>> things weeks is something I personally would like not buried so far > down > >>> in > >>> the thread). Totally my bad, sorry about that. > >>> > >>> Other than that, it sounds like we're pretty much in agreement. > >>> > >>> Here's my current understanding of the state and consensus as of right > >>> now > >>> (which is subject to change as more discussion happens): > >>> > >>> - Most of the people in the thread are in favor of #2 for 0.7.1 and #3 > >>> for 0.8.0. > >>> - I don't believe I've seen an explicit response from Otto on what > >>> he > >>> thinks about doing this, and from a personal perspective like to > >>> see what > >>> his opinion is as the person who originally brought it up. > >>> - Mike said he's going to kick out a PR that addresses #2 > >>> - After that undergoes the normal review process and is merged, we > >>> proceed normally and cut RC2. > >>> > >>> > >>> On Thu, May 2, 2019 at 1:14 PM Michael Miklavcic < > >>> michael.miklav...@gmail.com> wrote: > >>> > >>> > I think your later point about using a project release version, from > >>> the > >>> > example of using other projects, is a valid one. To exactly that > >>> point, a > >>> > community member (Otto) brought up an issue/bug they found through > >>> testing > >>> > that they were previously unaware of and did not find documented. > >>> Which was > >>> > argued would be confusing to someone wanting to use a published > >>> release. We > >>> > discussed the implications of this bug/feature gap. And incidentally, > >>> it > >>> > sounds like some people see full dev as more useful than just a dev > >>> box, > >>> > others do not, independent of what we chose to name it. That came > from > >>> our > >>> > discussion about it. > >>> > > >>> > The expectation I had from my discussion with the contributors was > that > >>> > this fix for aggregation
Re: [DISCUSS] Metron Release - 0.7.1 next steps
Starting the default parsers as an aggregated topology was introduced when we added support for the PCAP Topology [1], as a workaround to the limited supervisor slots available in full dev. At that point, the full effect of the change should have been determined. It also warranted a DISCUSS thread before we went ahead with the change. It was a mistake on my part and I apologize. In the interim, we could free up slots on the full dev by stopping lesser user topologies (e.g. PCAP) so that the default parsers can be started as individual topologies. This can be a temporary solution till such time the UI support for parser aggregation comes through. How does this sound? -Anand [1] https://tinyurl.com/y4yoeszo On 5/3/19, 4:23 AM, "Michael Miklavcic" wrote: Whether or not full dev is, first and foremost, "dev" I think your questions being up a good point. If not full_dev for introducing new users, then what? If we want to provide a different env for letting people tinker and try it out than we do for development, that's completely fine. But we don't have that right now. So we can treat full_dev as multipurpose, or we can stop directing non-devs to it, or we can add something new. I honestly don't have any recommendations here. We've talked about docker instances for replacing in-memory components, but I'm still not sure that solves this problem, or adds more complexity. Given the current options on the table, I'm inclined to go with "full_dev" serves both dev and demo purposes. Otto, what do you think? On Thu, May 2, 2019, 4:32 PM Otto Fowler wrote: > I’ve commented on the PR, and I won’t repeat it here as well, I will > however ask again if we know and can list all of the usability issues that > surround this problem. IE. All the things that can happen or may happen > for people who are not Metron developers and committers who are using > full dev, because we keep recommending it. > > > > On May 2, 2019 at 17:38:30, Michael Miklavcic (michael.miklav...@gmail.com > ) > wrote: > > PR is up. I added the doc change to the metron-deployment README since this > serves as the gateway doc for all the VM instances. All of which would be > affected by the feature gap. > > https://github.com/apache/metron/pull/1398 > > On Thu, May 2, 2019 at 1:37 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Here's the ticket I created to track it, which also references the Jira > > for the new UI feature. > > https://issues.apache.org/jira/browse/METRON-2100 > > > > On Thu, May 2, 2019 at 12:34 PM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > >> :-) > >> > >> I expect to have #2 out sometime today. > >> > >> On Thu, May 2, 2019, 12:11 PM Justin Leet > wrote: > >> > >>> > > >>> > I personally > >>> > don't like this feature gap in full dev. It seems Otto agrees, and > >>> Casey at > >>> > the very least sees it as enough of an issue to gate us from 0.8. > >>> > > >>> > >>> +1 on all of this. I don't like it either. > >>> > >>> > >>> > Our vote landed 2-2. We are having a discussion about what to do with > >>> the > >>> > release. This is that discussion. > >>> > >>> > >>> I'm going to be honest, my response was a combination of misreading > what > >>> you said and thinking you were proposing delaying the release more > >>> seriously and feeling a bit blindsided by a perceived move from the > >>> initial > >>> "take more time than originally anticipated" (which in my head I took > as > >>> a > >>> couple days) to "versus next week, or the week after" (where delaying > >>> things weeks is something I personally would like not buried so far > down > >>> in > >>> the thread). Totally my bad, sorry about that. > >>> > >>> Other than that, it sounds like we're pretty much in agreement. > >>> > >>> Here's my current understanding of the state and consensus as of right > >>> now > >>> (which is subject to change as more discussion happens): > >>> > >>> - Most of the people in the thread are in favor of #2 for 0.7.1 and #3 > >>> for 0.8.0. > >>> - I don't believe I've seen an explicit response from Otto on what > >>> he > >>> thinks about doing this, and from a personal perspective like to > >>> see what > >>> his opinion is as the person who originally brought it up. > >>> - Mike said he's going to kick out a PR that addresses #2 > >>> - After that undergoes the normal review process and is merged, we > >>> proceed normally and cut RC2. > >>> > >>> > >>> On Thu, May 2, 2019 at 1:14 PM Michael Miklavcic < > >>> michael.miklav...@gmail.com> wrote: > >>> > >>> > I think your later point about
Re: [DISCUSS] Metron Release - 0.7.1 next steps
Whether or not full dev is, first and foremost, "dev" I think your questions being up a good point. If not full_dev for introducing new users, then what? If we want to provide a different env for letting people tinker and try it out than we do for development, that's completely fine. But we don't have that right now. So we can treat full_dev as multipurpose, or we can stop directing non-devs to it, or we can add something new. I honestly don't have any recommendations here. We've talked about docker instances for replacing in-memory components, but I'm still not sure that solves this problem, or adds more complexity. Given the current options on the table, I'm inclined to go with "full_dev" serves both dev and demo purposes. Otto, what do you think? On Thu, May 2, 2019, 4:32 PM Otto Fowler wrote: > I’ve commented on the PR, and I won’t repeat it here as well, I will > however ask again if we know and can list all of the usability issues that > surround this problem. IE. All the things that can happen or may happen > for people who are not Metron developers and committers who are using > full dev, because we keep recommending it. > > > > On May 2, 2019 at 17:38:30, Michael Miklavcic (michael.miklav...@gmail.com > ) > wrote: > > PR is up. I added the doc change to the metron-deployment README since this > serves as the gateway doc for all the VM instances. All of which would be > affected by the feature gap. > > https://github.com/apache/metron/pull/1398 > > On Thu, May 2, 2019 at 1:37 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Here's the ticket I created to track it, which also references the Jira > > for the new UI feature. > > https://issues.apache.org/jira/browse/METRON-2100 > > > > On Thu, May 2, 2019 at 12:34 PM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > >> :-) > >> > >> I expect to have #2 out sometime today. > >> > >> On Thu, May 2, 2019, 12:11 PM Justin Leet > wrote: > >> > >>> > > >>> > I personally > >>> > don't like this feature gap in full dev. It seems Otto agrees, and > >>> Casey at > >>> > the very least sees it as enough of an issue to gate us from 0.8. > >>> > > >>> > >>> +1 on all of this. I don't like it either. > >>> > >>> > >>> > Our vote landed 2-2. We are having a discussion about what to do with > >>> the > >>> > release. This is that discussion. > >>> > >>> > >>> I'm going to be honest, my response was a combination of misreading > what > >>> you said and thinking you were proposing delaying the release more > >>> seriously and feeling a bit blindsided by a perceived move from the > >>> initial > >>> "take more time than originally anticipated" (which in my head I took > as > >>> a > >>> couple days) to "versus next week, or the week after" (where delaying > >>> things weeks is something I personally would like not buried so far > down > >>> in > >>> the thread). Totally my bad, sorry about that. > >>> > >>> Other than that, it sounds like we're pretty much in agreement. > >>> > >>> Here's my current understanding of the state and consensus as of right > >>> now > >>> (which is subject to change as more discussion happens): > >>> > >>> - Most of the people in the thread are in favor of #2 for 0.7.1 and #3 > >>> for 0.8.0. > >>> - I don't believe I've seen an explicit response from Otto on what > >>> he > >>> thinks about doing this, and from a personal perspective like to > >>> see what > >>> his opinion is as the person who originally brought it up. > >>> - Mike said he's going to kick out a PR that addresses #2 > >>> - After that undergoes the normal review process and is merged, we > >>> proceed normally and cut RC2. > >>> > >>> > >>> On Thu, May 2, 2019 at 1:14 PM Michael Miklavcic < > >>> michael.miklav...@gmail.com> wrote: > >>> > >>> > I think your later point about using a project release version, from > >>> the > >>> > example of using other projects, is a valid one. To exactly that > >>> point, a > >>> > community member (Otto) brought up an issue/bug they found through > >>> testing > >>> > that they were previously unaware of and did not find documented. > >>> Which was > >>> > argued would be confusing to someone wanting to use a published > >>> release. We > >>> > discussed the implications of this bug/feature gap. And incidentally, > >>> it > >>> > sounds like some people see full dev as more useful than just a dev > >>> box, > >>> > others do not, independent of what we chose to name it. That came > from > >>> our > >>> > discussion about it. > >>> > > >>> > The expectation I had from my discussion with the contributors was > that > >>> > this fix for aggregation was ready. So to your point about whether it > >>> > belonged or not, I'm inclined to say yes, had it been ready. I > >>> personally > >>> > don't like this feature gap in full dev. It seems Otto agrees, and > >>> Casey at > >>> > the very least sees it as enough of an issue to gate us from 0.8. New > >>> > information about that feature has changed my
Re: [DISCUSS] Metron Release - 0.7.1 next steps
I’ve commented on the PR, and I won’t repeat it here as well, I will however ask again if we know and can list all of the usability issues that surround this problem. IE. All the things that can happen or may happen for people who are not Metron developers and committers who are using full dev, because we keep recommending it. On May 2, 2019 at 17:38:30, Michael Miklavcic (michael.miklav...@gmail.com) wrote: PR is up. I added the doc change to the metron-deployment README since this serves as the gateway doc for all the VM instances. All of which would be affected by the feature gap. https://github.com/apache/metron/pull/1398 On Thu, May 2, 2019 at 1:37 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Here's the ticket I created to track it, which also references the Jira > for the new UI feature. > https://issues.apache.org/jira/browse/METRON-2100 > > On Thu, May 2, 2019 at 12:34 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> :-) >> >> I expect to have #2 out sometime today. >> >> On Thu, May 2, 2019, 12:11 PM Justin Leet wrote: >> >>> > >>> > I personally >>> > don't like this feature gap in full dev. It seems Otto agrees, and >>> Casey at >>> > the very least sees it as enough of an issue to gate us from 0.8. >>> > >>> >>> +1 on all of this. I don't like it either. >>> >>> >>> > Our vote landed 2-2. We are having a discussion about what to do with >>> the >>> > release. This is that discussion. >>> >>> >>> I'm going to be honest, my response was a combination of misreading what >>> you said and thinking you were proposing delaying the release more >>> seriously and feeling a bit blindsided by a perceived move from the >>> initial >>> "take more time than originally anticipated" (which in my head I took as >>> a >>> couple days) to "versus next week, or the week after" (where delaying >>> things weeks is something I personally would like not buried so far down >>> in >>> the thread). Totally my bad, sorry about that. >>> >>> Other than that, it sounds like we're pretty much in agreement. >>> >>> Here's my current understanding of the state and consensus as of right >>> now >>> (which is subject to change as more discussion happens): >>> >>> - Most of the people in the thread are in favor of #2 for 0.7.1 and #3 >>> for 0.8.0. >>> - I don't believe I've seen an explicit response from Otto on what >>> he >>> thinks about doing this, and from a personal perspective like to >>> see what >>> his opinion is as the person who originally brought it up. >>> - Mike said he's going to kick out a PR that addresses #2 >>> - After that undergoes the normal review process and is merged, we >>> proceed normally and cut RC2. >>> >>> >>> On Thu, May 2, 2019 at 1:14 PM Michael Miklavcic < >>> michael.miklav...@gmail.com> wrote: >>> >>> > I think your later point about using a project release version, from >>> the >>> > example of using other projects, is a valid one. To exactly that >>> point, a >>> > community member (Otto) brought up an issue/bug they found through >>> testing >>> > that they were previously unaware of and did not find documented. >>> Which was >>> > argued would be confusing to someone wanting to use a published >>> release. We >>> > discussed the implications of this bug/feature gap. And incidentally, >>> it >>> > sounds like some people see full dev as more useful than just a dev >>> box, >>> > others do not, independent of what we chose to name it. That came from >>> our >>> > discussion about it. >>> > >>> > The expectation I had from my discussion with the contributors was that >>> > this fix for aggregation was ready. So to your point about whether it >>> > belonged or not, I'm inclined to say yes, had it been ready. I >>> personally >>> > don't like this feature gap in full dev. It seems Otto agrees, and >>> Casey at >>> > the very least sees it as enough of an issue to gate us from 0.8. New >>> > information about that feature has changed my mind about what to do >>> about >>> > it in the short term. I think we should move forward. >>> > >>> > Our vote landed 2-2. We are having a discussion about what to do with >>> the >>> > release. This is that discussion. >>> > >>> > On Thu, May 2, 2019, 10:52 AM Justin Leet >>> wrote: >>> > >>> > > @Mike >>> > > I have a different question: Why is this enough to consider delaying >>> a >>> > > release in the first place for a fairly involved fix? >>> > > >>> > > There was a discuss thread, where the general agreement was that we >>> had >>> > > enough value to do a release (Over a month ago. And more things have >>> gone >>> > > into master since then). There's a good number of fixes, and not just >>> > > trivial ones either. The general consensus here seems to be that the >>> > > management UI issue is fairly minor for a point release (after all, >>> > there's >>> > > been multiple people who think option 2 is sufficient), but becomes >>> > > important if we want to release a minor version. The question I asked >>>
Re: [DISCUSS] Metron Release - 0.7.1 next steps
> As a separate issue, I will also volunteer to see if I can help Tamas find the discuss thread mentioned. It should be linked to the PR or feature branch for reference. That may also be a gap in dev guidelines that should be spelled out. Just following up on this. I checked and verified we have existing recommendations around discussion threads for new features, so no immediate tasks to tackle there - https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines " 1.1 Contributing A Code Change ... If you are introducing a completely new feature or API it is a good idea to start a discussion and get consensus on the basic design first. Larger changes should be discussed on the dev boards before submission. New features and significant bug fixes should be documented in the JIRA and appropriate architecture diagrams should be attached. Major features may require a vote. Note that if the change is related to user-facing protocols / interface / configs, etc, you need to make the corresponding change on the documentation as well. " Per the new DISCUSS thread and Jira updates from Shane ( https://lists.apache.org/thread.html/ad206bdd59594cf74560770dfdbfcde0addd120d6fa8ea73f1a92a6b@%3Cdev.metron.apache.org%3E) it looks like we're in good shape for referencing past discussions for this feature and have any remaining gaps being covered. On Thu, May 2, 2019 at 7:31 AM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > I am still in favor of option 2. I will volunteer and submit the doc PR. I > agree we should not rush through a review process for a maintenance > release. The implications to the UI, as Otto asked, are that aggregated > parsers will not show up in the UI. You cannot create them there. Actually, > any parser not created through the UI (eg CLI) will not show up in the UI, > aggregated or not. > > As a separate issue, I will also volunteer to see if I can help Tamas find > the discuss thread mentioned. It should be linked to the PR or feature > branch for reference. That may also be a gap in dev guidelines that should > be spelled out. > > On Thu, May 2, 2019, 7:17 AM Nick Allen wrote: > >> To echo Justin's comments, I am in favor of #2, which provides a clear, >> well-defined path to a release. >> >>- Why hold back a release, especially a point release containing 89 >>improvements, for one issue that will not affect most users? >> >> >>- It is one thing to stall a release to address a bug of limited scope, >>where a fix is well understood and ready for review, but it is >> completely >>another issue to delay for this. >> >> >>- I don't see a set of reviewable PRs yet that will push this over the >>finish line. As has been noted, there were fundamental problems with >> #1360 >>(which has now been closed) that would have prevented adequate review >> by >>the community. >> >> >>- Why drive this issue with the pressure of a stalled release, instead >>of just releasing the fix when it is ready and has been adequately >>reviewed? Swarming on an issue does not often produce quality results. >> >> For those in favor of #1, can someone please provide a clear outline of >> what the fix looks-like? How many PRs will this require? When are these >> PRs likely to be ready? Who is driving this? Tamás has already commented >> that this not a quick fix. This path is very murky to me, but maybe I am >> just ignorant on this. >> >> I would also urge other committers and users who don't have a binding vote >> on the release to share their opinion on the path forward. >> >> >> >> >> On Thu, May 2, 2019 at 7:17 AM Otto Fowler >> wrote: >> >> > If you can find a link in the archives for that thread, it would really >> > help. >> > >> > I don’t think sending them up as one sensor would work…. as something >> > quick. I think it is an interesting idea from a higher level that would >> > need some more thought though ( IE: what if every sensor in the ui was a >> > sensor group, and the existing entries where just groups of 1 ). >> > >> > As far as I can see, we have brought up the idea of a release >> ourselves, I >> > don’t see why we don’t just swarm this issue and get it right then >> release. >> > >> > >> > >> > On May 2, 2019 at 04:16:31, Tamás Fodor (ftamas.m...@gmail.com) wrote: >> > >> > In PR#1360 we introduced a new state management strategy involving a new >> > module called Ngrx. We had a discussion thread on this a few months ago >> and >> > we successfully convinced you about the benefits. This is one of the >> > reasons why this PR is going to be still huge after cleaning up the >> commit >> > history. After you having a look at the changes and the feature itself, >> > there's likely have questions about why certain parts work as they do. >> The >> > thing what I'd like to point out is that, yes, it probably takes more >> time >> > to get it in. >> > >> > In order to being able to release the RC, wouldn't it be an easy and >>
Re: [DISCUSS] Metron Release - 0.7.1 next steps
PR is up. I added the doc change to the metron-deployment README since this serves as the gateway doc for all the VM instances. All of which would be affected by the feature gap. https://github.com/apache/metron/pull/1398 On Thu, May 2, 2019 at 1:37 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Here's the ticket I created to track it, which also references the Jira > for the new UI feature. > https://issues.apache.org/jira/browse/METRON-2100 > > On Thu, May 2, 2019 at 12:34 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> :-) >> >> I expect to have #2 out sometime today. >> >> On Thu, May 2, 2019, 12:11 PM Justin Leet wrote: >> >>> > >>> > I personally >>> > don't like this feature gap in full dev. It seems Otto agrees, and >>> Casey at >>> > the very least sees it as enough of an issue to gate us from 0.8. >>> > >>> >>> +1 on all of this. I don't like it either. >>> >>> >>> > Our vote landed 2-2. We are having a discussion about what to do with >>> the >>> > release. This is that discussion. >>> >>> >>> I'm going to be honest, my response was a combination of misreading what >>> you said and thinking you were proposing delaying the release more >>> seriously and feeling a bit blindsided by a perceived move from the >>> initial >>> "take more time than originally anticipated" (which in my head I took as >>> a >>> couple days) to "versus next week, or the week after" (where delaying >>> things weeks is something I personally would like not buried so far down >>> in >>> the thread). Totally my bad, sorry about that. >>> >>> Other than that, it sounds like we're pretty much in agreement. >>> >>> Here's my current understanding of the state and consensus as of right >>> now >>> (which is subject to change as more discussion happens): >>> >>>- Most of the people in the thread are in favor of #2 for 0.7.1 and #3 >>>for 0.8.0. >>> - I don't believe I've seen an explicit response from Otto on what >>> he >>> thinks about doing this, and from a personal perspective like to >>> see what >>> his opinion is as the person who originally brought it up. >>>- Mike said he's going to kick out a PR that addresses #2 >>>- After that undergoes the normal review process and is merged, we >>>proceed normally and cut RC2. >>> >>> >>> On Thu, May 2, 2019 at 1:14 PM Michael Miklavcic < >>> michael.miklav...@gmail.com> wrote: >>> >>> > I think your later point about using a project release version, from >>> the >>> > example of using other projects, is a valid one. To exactly that >>> point, a >>> > community member (Otto) brought up an issue/bug they found through >>> testing >>> > that they were previously unaware of and did not find documented. >>> Which was >>> > argued would be confusing to someone wanting to use a published >>> release. We >>> > discussed the implications of this bug/feature gap. And incidentally, >>> it >>> > sounds like some people see full dev as more useful than just a dev >>> box, >>> > others do not, independent of what we chose to name it. That came from >>> our >>> > discussion about it. >>> > >>> > The expectation I had from my discussion with the contributors was that >>> > this fix for aggregation was ready. So to your point about whether it >>> > belonged or not, I'm inclined to say yes, had it been ready. I >>> personally >>> > don't like this feature gap in full dev. It seems Otto agrees, and >>> Casey at >>> > the very least sees it as enough of an issue to gate us from 0.8. New >>> > information about that feature has changed my mind about what to do >>> about >>> > it in the short term. I think we should move forward. >>> > >>> > Our vote landed 2-2. We are having a discussion about what to do with >>> the >>> > release. This is that discussion. >>> > >>> > On Thu, May 2, 2019, 10:52 AM Justin Leet >>> wrote: >>> > >>> > > @Mike >>> > > I have a different question: Why is this enough to consider delaying >>> a >>> > > release in the first place for a fairly involved fix? >>> > > >>> > > There was a discuss thread, where the general agreement was that we >>> had >>> > > enough value to do a release (Over a month ago. And more things have >>> gone >>> > > into master since then). There's a good number of fixes, and not just >>> > > trivial ones either. The general consensus here seems to be that the >>> > > management UI issue is fairly minor for a point release (after all, >>> > there's >>> > > been multiple people who think option 2 is sufficient), but becomes >>> > > important if we want to release a minor version. The question I asked >>> > > myself about this was ""Does this issue detract enough value that a >>> > release >>> > > isn't worthwhile?" and my answer was, and still is, "No, we have >>> enough >>> > > value to do a meaningful release". >>> > > >>> > > I'm fine with delaying or cancelling a release because we find issues >>> > that >>> > > are severe enough or we don't think there's enough value anymore,
Re: [DISCUSS] Metron Release - 0.7.1 next steps
Here's the ticket I created to track it, which also references the Jira for the new UI feature. https://issues.apache.org/jira/browse/METRON-2100 On Thu, May 2, 2019 at 12:34 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > :-) > > I expect to have #2 out sometime today. > > On Thu, May 2, 2019, 12:11 PM Justin Leet wrote: > >> > >> > I personally >> > don't like this feature gap in full dev. It seems Otto agrees, and >> Casey at >> > the very least sees it as enough of an issue to gate us from 0.8. >> > >> >> +1 on all of this. I don't like it either. >> >> >> > Our vote landed 2-2. We are having a discussion about what to do with >> the >> > release. This is that discussion. >> >> >> I'm going to be honest, my response was a combination of misreading what >> you said and thinking you were proposing delaying the release more >> seriously and feeling a bit blindsided by a perceived move from the >> initial >> "take more time than originally anticipated" (which in my head I took as a >> couple days) to "versus next week, or the week after" (where delaying >> things weeks is something I personally would like not buried so far down >> in >> the thread). Totally my bad, sorry about that. >> >> Other than that, it sounds like we're pretty much in agreement. >> >> Here's my current understanding of the state and consensus as of right now >> (which is subject to change as more discussion happens): >> >>- Most of the people in the thread are in favor of #2 for 0.7.1 and #3 >>for 0.8.0. >> - I don't believe I've seen an explicit response from Otto on what >> he >> thinks about doing this, and from a personal perspective like to >> see what >> his opinion is as the person who originally brought it up. >>- Mike said he's going to kick out a PR that addresses #2 >>- After that undergoes the normal review process and is merged, we >>proceed normally and cut RC2. >> >> >> On Thu, May 2, 2019 at 1:14 PM Michael Miklavcic < >> michael.miklav...@gmail.com> wrote: >> >> > I think your later point about using a project release version, from the >> > example of using other projects, is a valid one. To exactly that >> point, a >> > community member (Otto) brought up an issue/bug they found through >> testing >> > that they were previously unaware of and did not find documented. Which >> was >> > argued would be confusing to someone wanting to use a published >> release. We >> > discussed the implications of this bug/feature gap. And incidentally, it >> > sounds like some people see full dev as more useful than just a dev box, >> > others do not, independent of what we chose to name it. That came from >> our >> > discussion about it. >> > >> > The expectation I had from my discussion with the contributors was that >> > this fix for aggregation was ready. So to your point about whether it >> > belonged or not, I'm inclined to say yes, had it been ready. I >> personally >> > don't like this feature gap in full dev. It seems Otto agrees, and >> Casey at >> > the very least sees it as enough of an issue to gate us from 0.8. New >> > information about that feature has changed my mind about what to do >> about >> > it in the short term. I think we should move forward. >> > >> > Our vote landed 2-2. We are having a discussion about what to do with >> the >> > release. This is that discussion. >> > >> > On Thu, May 2, 2019, 10:52 AM Justin Leet >> wrote: >> > >> > > @Mike >> > > I have a different question: Why is this enough to consider delaying a >> > > release in the first place for a fairly involved fix? >> > > >> > > There was a discuss thread, where the general agreement was that we >> had >> > > enough value to do a release (Over a month ago. And more things have >> gone >> > > into master since then). There's a good number of fixes, and not just >> > > trivial ones either. The general consensus here seems to be that the >> > > management UI issue is fairly minor for a point release (after all, >> > there's >> > > been multiple people who think option 2 is sufficient), but becomes >> > > important if we want to release a minor version. The question I asked >> > > myself about this was ""Does this issue detract enough value that a >> > release >> > > isn't worthwhile?" and my answer was, and still is, "No, we have >> enough >> > > value to do a meaningful release". >> > > >> > > I'm fine with delaying or cancelling a release because we find issues >> > that >> > > are severe enough or we don't think there's enough value anymore, but >> to >> > be >> > > entirely honest, I'm absolutely shocked this issue has blown up so >> much. >> > > However, if you want to have a discuss thread to reevaluate if it's >> > > worthwhile to do a release, go for it. The communities' calculus on >> the >> > > "Does this issue detract enough value that a release isn't >> worthwhile?" >> > may >> > > be different than mine. >> > > >> > > Having said all that, to a large extent, I think you're
Re: [DISCUSS] Metron Release - 0.7.1 next steps
:-) I expect to have #2 out sometime today. On Thu, May 2, 2019, 12:11 PM Justin Leet wrote: > > > > I personally > > don't like this feature gap in full dev. It seems Otto agrees, and Casey > at > > the very least sees it as enough of an issue to gate us from 0.8. > > > > +1 on all of this. I don't like it either. > > > > Our vote landed 2-2. We are having a discussion about what to do with the > > release. This is that discussion. > > > I'm going to be honest, my response was a combination of misreading what > you said and thinking you were proposing delaying the release more > seriously and feeling a bit blindsided by a perceived move from the initial > "take more time than originally anticipated" (which in my head I took as a > couple days) to "versus next week, or the week after" (where delaying > things weeks is something I personally would like not buried so far down in > the thread). Totally my bad, sorry about that. > > Other than that, it sounds like we're pretty much in agreement. > > Here's my current understanding of the state and consensus as of right now > (which is subject to change as more discussion happens): > >- Most of the people in the thread are in favor of #2 for 0.7.1 and #3 >for 0.8.0. > - I don't believe I've seen an explicit response from Otto on what he > thinks about doing this, and from a personal perspective like to see > what > his opinion is as the person who originally brought it up. >- Mike said he's going to kick out a PR that addresses #2 >- After that undergoes the normal review process and is merged, we >proceed normally and cut RC2. > > > On Thu, May 2, 2019 at 1:14 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > I think your later point about using a project release version, from the > > example of using other projects, is a valid one. To exactly that point, > a > > community member (Otto) brought up an issue/bug they found through > testing > > that they were previously unaware of and did not find documented. Which > was > > argued would be confusing to someone wanting to use a published release. > We > > discussed the implications of this bug/feature gap. And incidentally, it > > sounds like some people see full dev as more useful than just a dev box, > > others do not, independent of what we chose to name it. That came from > our > > discussion about it. > > > > The expectation I had from my discussion with the contributors was that > > this fix for aggregation was ready. So to your point about whether it > > belonged or not, I'm inclined to say yes, had it been ready. I personally > > don't like this feature gap in full dev. It seems Otto agrees, and Casey > at > > the very least sees it as enough of an issue to gate us from 0.8. New > > information about that feature has changed my mind about what to do about > > it in the short term. I think we should move forward. > > > > Our vote landed 2-2. We are having a discussion about what to do with the > > release. This is that discussion. > > > > On Thu, May 2, 2019, 10:52 AM Justin Leet wrote: > > > > > @Mike > > > I have a different question: Why is this enough to consider delaying a > > > release in the first place for a fairly involved fix? > > > > > > There was a discuss thread, where the general agreement was that we had > > > enough value to do a release (Over a month ago. And more things have > gone > > > into master since then). There's a good number of fixes, and not just > > > trivial ones either. The general consensus here seems to be that the > > > management UI issue is fairly minor for a point release (after all, > > there's > > > been multiple people who think option 2 is sufficient), but becomes > > > important if we want to release a minor version. The question I asked > > > myself about this was ""Does this issue detract enough value that a > > release > > > isn't worthwhile?" and my answer was, and still is, "No, we have enough > > > value to do a meaningful release". > > > > > > I'm fine with delaying or cancelling a release because we find issues > > that > > > are severe enough or we don't think there's enough value anymore, but > to > > be > > > entirely honest, I'm absolutely shocked this issue has blown up so > much. > > > However, if you want to have a discuss thread to reevaluate if it's > > > worthwhile to do a release, go for it. The communities' calculus on > the > > > "Does this issue detract enough value that a release isn't worthwhile?" > > may > > > be different than mine. > > > > > > Having said all that, to a large extent, I think you're right. It > really > > > doesn't matter* that much* if we release next week or the week after or > > > whenever. But at the same time I personally get super frustrated when I > > go > > > to use a project, find a bug, it's already known and fixed, but it just > > > never puts out a released version. Every cutoff is largely arbitrary, > > but > > > I think getting our improvements
Re: [DISCUSS] Metron Release - 0.7.1 next steps
> > I personally > don't like this feature gap in full dev. It seems Otto agrees, and Casey at > the very least sees it as enough of an issue to gate us from 0.8. > +1 on all of this. I don't like it either. > Our vote landed 2-2. We are having a discussion about what to do with the > release. This is that discussion. I'm going to be honest, my response was a combination of misreading what you said and thinking you were proposing delaying the release more seriously and feeling a bit blindsided by a perceived move from the initial "take more time than originally anticipated" (which in my head I took as a couple days) to "versus next week, or the week after" (where delaying things weeks is something I personally would like not buried so far down in the thread). Totally my bad, sorry about that. Other than that, it sounds like we're pretty much in agreement. Here's my current understanding of the state and consensus as of right now (which is subject to change as more discussion happens): - Most of the people in the thread are in favor of #2 for 0.7.1 and #3 for 0.8.0. - I don't believe I've seen an explicit response from Otto on what he thinks about doing this, and from a personal perspective like to see what his opinion is as the person who originally brought it up. - Mike said he's going to kick out a PR that addresses #2 - After that undergoes the normal review process and is merged, we proceed normally and cut RC2. On Thu, May 2, 2019 at 1:14 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > I think your later point about using a project release version, from the > example of using other projects, is a valid one. To exactly that point, a > community member (Otto) brought up an issue/bug they found through testing > that they were previously unaware of and did not find documented. Which was > argued would be confusing to someone wanting to use a published release. We > discussed the implications of this bug/feature gap. And incidentally, it > sounds like some people see full dev as more useful than just a dev box, > others do not, independent of what we chose to name it. That came from our > discussion about it. > > The expectation I had from my discussion with the contributors was that > this fix for aggregation was ready. So to your point about whether it > belonged or not, I'm inclined to say yes, had it been ready. I personally > don't like this feature gap in full dev. It seems Otto agrees, and Casey at > the very least sees it as enough of an issue to gate us from 0.8. New > information about that feature has changed my mind about what to do about > it in the short term. I think we should move forward. > > Our vote landed 2-2. We are having a discussion about what to do with the > release. This is that discussion. > > On Thu, May 2, 2019, 10:52 AM Justin Leet wrote: > > > @Mike > > I have a different question: Why is this enough to consider delaying a > > release in the first place for a fairly involved fix? > > > > There was a discuss thread, where the general agreement was that we had > > enough value to do a release (Over a month ago. And more things have gone > > into master since then). There's a good number of fixes, and not just > > trivial ones either. The general consensus here seems to be that the > > management UI issue is fairly minor for a point release (after all, > there's > > been multiple people who think option 2 is sufficient), but becomes > > important if we want to release a minor version. The question I asked > > myself about this was ""Does this issue detract enough value that a > release > > isn't worthwhile?" and my answer was, and still is, "No, we have enough > > value to do a meaningful release". > > > > I'm fine with delaying or cancelling a release because we find issues > that > > are severe enough or we don't think there's enough value anymore, but to > be > > entirely honest, I'm absolutely shocked this issue has blown up so much. > > However, if you want to have a discuss thread to reevaluate if it's > > worthwhile to do a release, go for it. The communities' calculus on the > > "Does this issue detract enough value that a release isn't worthwhile?" > may > > be different than mine. > > > > Having said all that, to a large extent, I think you're right. It really > > doesn't matter* that much* if we release next week or the week after or > > whenever. But at the same time I personally get super frustrated when I > go > > to use a project, find a bug, it's already known and fixed, but it just > > never puts out a released version. Every cutoff is largely arbitrary, > but > > I think getting our improvements and fixes out there is important. One of > > the things we've done fairly well is put out releases at a fairly decent > > cadence for a project this large. I really don't want to set the > precedent > > of just increasingly pushing out point releases for stuff like this. > > > > On Thu, May 2, 2019 at 12:52
Re: [DISCUSS] Metron Release - 0.7.1 next steps
I think your later point about using a project release version, from the example of using other projects, is a valid one. To exactly that point, a community member (Otto) brought up an issue/bug they found through testing that they were previously unaware of and did not find documented. Which was argued would be confusing to someone wanting to use a published release. We discussed the implications of this bug/feature gap. And incidentally, it sounds like some people see full dev as more useful than just a dev box, others do not, independent of what we chose to name it. That came from our discussion about it. The expectation I had from my discussion with the contributors was that this fix for aggregation was ready. So to your point about whether it belonged or not, I'm inclined to say yes, had it been ready. I personally don't like this feature gap in full dev. It seems Otto agrees, and Casey at the very least sees it as enough of an issue to gate us from 0.8. New information about that feature has changed my mind about what to do about it in the short term. I think we should move forward. Our vote landed 2-2. We are having a discussion about what to do with the release. This is that discussion. On Thu, May 2, 2019, 10:52 AM Justin Leet wrote: > @Mike > I have a different question: Why is this enough to consider delaying a > release in the first place for a fairly involved fix? > > There was a discuss thread, where the general agreement was that we had > enough value to do a release (Over a month ago. And more things have gone > into master since then). There's a good number of fixes, and not just > trivial ones either. The general consensus here seems to be that the > management UI issue is fairly minor for a point release (after all, there's > been multiple people who think option 2 is sufficient), but becomes > important if we want to release a minor version. The question I asked > myself about this was ""Does this issue detract enough value that a release > isn't worthwhile?" and my answer was, and still is, "No, we have enough > value to do a meaningful release". > > I'm fine with delaying or cancelling a release because we find issues that > are severe enough or we don't think there's enough value anymore, but to be > entirely honest, I'm absolutely shocked this issue has blown up so much. > However, if you want to have a discuss thread to reevaluate if it's > worthwhile to do a release, go for it. The communities' calculus on the > "Does this issue detract enough value that a release isn't worthwhile?" may > be different than mine. > > Having said all that, to a large extent, I think you're right. It really > doesn't matter* that much* if we release next week or the week after or > whenever. But at the same time I personally get super frustrated when I go > to use a project, find a bug, it's already known and fixed, but it just > never puts out a released version. Every cutoff is largely arbitrary, but > I think getting our improvements and fixes out there is important. One of > the things we've done fairly well is put out releases at a fairly decent > cadence for a project this large. I really don't want to set the precedent > of just increasingly pushing out point releases for stuff like this. > > On Thu, May 2, 2019 at 12:52 PM Nick Allen wrote: > > > I think any open source project needs to strive to cut releases > regularly. > > This is healthy for the project and community. It gets new features and > > functionality out to the community so we can get feedback, find what is > > working and what is not, iterate and improve. You probably agree with > > this. > > > > While releasing this week or next may not matter in the grand scheme, if > we > > want to cut releases regularly, then we need to bear down and just do it. > > Case in point, I opened the initial discussion for this release on March > > 13th [1] and it is now May 2nd and we have yet to release 7 weeks later. > > > > -- > > [1] > > > > > https://lists.apache.org/thread.html/4f58649139f0aa6276f96febe1d0ecf9e6b3fb5b2b088cba1e3c4d81@%3Cdev.metron.apache.org%3E > > > > > > On Thu, May 2, 2019 at 11:51 AM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > > > As a more general question, can I ask why we're feeling pressure to > push > > > out a release in the first place? Again, I'm happy to continue with > > option > > > 2. Let's move forward and get out the release. But is there a reason > why > > we > > > think it has to get out now, versus next week, or the week after? Otto > > > pointed out a legitimate issue, dev environment or not, and I'm unclear > > why > > > we have an issue with waiting for the fix. There's no pressure on this, > > > imho. > > > > > > On Thu, May 2, 2019, 9:12 AM Otto Fowler > > wrote: > > > > > > > I remember this now, but I’m not sure how I would have related this > to > > a > > > > parser aggregation pr honestly. > > > > > > > > > > > > On May 2, 2019 at 07:54:13, Shane Ardell
Re: [DISCUSS] Metron Release - 0.7.1 next steps
@Mike I have a different question: Why is this enough to consider delaying a release in the first place for a fairly involved fix? There was a discuss thread, where the general agreement was that we had enough value to do a release (Over a month ago. And more things have gone into master since then). There's a good number of fixes, and not just trivial ones either. The general consensus here seems to be that the management UI issue is fairly minor for a point release (after all, there's been multiple people who think option 2 is sufficient), but becomes important if we want to release a minor version. The question I asked myself about this was ""Does this issue detract enough value that a release isn't worthwhile?" and my answer was, and still is, "No, we have enough value to do a meaningful release". I'm fine with delaying or cancelling a release because we find issues that are severe enough or we don't think there's enough value anymore, but to be entirely honest, I'm absolutely shocked this issue has blown up so much. However, if you want to have a discuss thread to reevaluate if it's worthwhile to do a release, go for it. The communities' calculus on the "Does this issue detract enough value that a release isn't worthwhile?" may be different than mine. Having said all that, to a large extent, I think you're right. It really doesn't matter* that much* if we release next week or the week after or whenever. But at the same time I personally get super frustrated when I go to use a project, find a bug, it's already known and fixed, but it just never puts out a released version. Every cutoff is largely arbitrary, but I think getting our improvements and fixes out there is important. One of the things we've done fairly well is put out releases at a fairly decent cadence for a project this large. I really don't want to set the precedent of just increasingly pushing out point releases for stuff like this. On Thu, May 2, 2019 at 12:52 PM Nick Allen wrote: > I think any open source project needs to strive to cut releases regularly. > This is healthy for the project and community. It gets new features and > functionality out to the community so we can get feedback, find what is > working and what is not, iterate and improve. You probably agree with > this. > > While releasing this week or next may not matter in the grand scheme, if we > want to cut releases regularly, then we need to bear down and just do it. > Case in point, I opened the initial discussion for this release on March > 13th [1] and it is now May 2nd and we have yet to release 7 weeks later. > > -- > [1] > > https://lists.apache.org/thread.html/4f58649139f0aa6276f96febe1d0ecf9e6b3fb5b2b088cba1e3c4d81@%3Cdev.metron.apache.org%3E > > > On Thu, May 2, 2019 at 11:51 AM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > As a more general question, can I ask why we're feeling pressure to push > > out a release in the first place? Again, I'm happy to continue with > option > > 2. Let's move forward and get out the release. But is there a reason why > we > > think it has to get out now, versus next week, or the week after? Otto > > pointed out a legitimate issue, dev environment or not, and I'm unclear > why > > we have an issue with waiting for the fix. There's no pressure on this, > > imho. > > > > On Thu, May 2, 2019, 9:12 AM Otto Fowler > wrote: > > > > > I remember this now, but I’m not sure how I would have related this to > a > > > parser aggregation pr honestly. > > > > > > > > > On May 2, 2019 at 07:54:13, Shane Ardell (shane.m.ard...@gmail.com) > > wrote: > > > > > > Here's a link to the ngrx discussion thread from a few months back: > > > > > > > > > https://lists.apache.org/thread.html/06a59ea42e8d9a9dea5f90aab4011e44434555f8b7f3cf21297c7c87@%3Cdev.metron.apache.org%3E > > > > > > On Thu, May 2, 2019 at 1:17 PM Otto Fowler > > > wrote: > > > > > > > If you can find a link in the archives for that thread, it would > really > > > > help. > > > > > > > > I don’t think sending them up as one sensor would work…. as something > > > > quick. I think it is an interesting idea from a higher level that > would > > > > need some more thought though ( IE: what if every sensor in the ui > was > > a > > > > sensor group, and the existing entries where just groups of 1 ). > > > > > > > > As far as I can see, we have brought up the idea of a release > > ourselves, > > > I > > > > don’t see why we don’t just swarm this issue and get it right then > > > release. > > > > > > > > > > > > > > > > On May 2, 2019 at 04:16:31, Tamás Fodor (ftamas.m...@gmail.com) > wrote: > > > > > > > > In PR#1360 we introduced a new state management strategy involving a > > new > > > > module called Ngrx. We had a discussion thread on this a few months > ago > > > and > > > > we successfully convinced you about the benefits. This is one of the > > > > reasons why this PR is going to be still huge after cleaning up the > > > commit > > > > history. After you
Re: [DISCUSS] Metron Release - 0.7.1 next steps
I think any open source project needs to strive to cut releases regularly. This is healthy for the project and community. It gets new features and functionality out to the community so we can get feedback, find what is working and what is not, iterate and improve. You probably agree with this. While releasing this week or next may not matter in the grand scheme, if we want to cut releases regularly, then we need to bear down and just do it. Case in point, I opened the initial discussion for this release on March 13th [1] and it is now May 2nd and we have yet to release 7 weeks later. -- [1] https://lists.apache.org/thread.html/4f58649139f0aa6276f96febe1d0ecf9e6b3fb5b2b088cba1e3c4d81@%3Cdev.metron.apache.org%3E On Thu, May 2, 2019 at 11:51 AM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > As a more general question, can I ask why we're feeling pressure to push > out a release in the first place? Again, I'm happy to continue with option > 2. Let's move forward and get out the release. But is there a reason why we > think it has to get out now, versus next week, or the week after? Otto > pointed out a legitimate issue, dev environment or not, and I'm unclear why > we have an issue with waiting for the fix. There's no pressure on this, > imho. > > On Thu, May 2, 2019, 9:12 AM Otto Fowler wrote: > > > I remember this now, but I’m not sure how I would have related this to a > > parser aggregation pr honestly. > > > > > > On May 2, 2019 at 07:54:13, Shane Ardell (shane.m.ard...@gmail.com) > wrote: > > > > Here's a link to the ngrx discussion thread from a few months back: > > > > > https://lists.apache.org/thread.html/06a59ea42e8d9a9dea5f90aab4011e44434555f8b7f3cf21297c7c87@%3Cdev.metron.apache.org%3E > > > > On Thu, May 2, 2019 at 1:17 PM Otto Fowler > > wrote: > > > > > If you can find a link in the archives for that thread, it would really > > > help. > > > > > > I don’t think sending them up as one sensor would work…. as something > > > quick. I think it is an interesting idea from a higher level that would > > > need some more thought though ( IE: what if every sensor in the ui was > a > > > sensor group, and the existing entries where just groups of 1 ). > > > > > > As far as I can see, we have brought up the idea of a release > ourselves, > > I > > > don’t see why we don’t just swarm this issue and get it right then > > release. > > > > > > > > > > > > On May 2, 2019 at 04:16:31, Tamás Fodor (ftamas.m...@gmail.com) wrote: > > > > > > In PR#1360 we introduced a new state management strategy involving a > new > > > module called Ngrx. We had a discussion thread on this a few months ago > > and > > > we successfully convinced you about the benefits. This is one of the > > > reasons why this PR is going to be still huge after cleaning up the > > commit > > > history. After you having a look at the changes and the feature itself, > > > there's likely have questions about why certain parts work as they do. > > The > > > thing what I'd like to point out is that, yes, it probably takes more > > time > > > to get it in. > > > > > > In order to being able to release the RC, wouldn't it be an easy and > > quick > > > fix on the backend if it sent the aggregated parsers to the client as > > they > > > were one sensor? It's just an idea, it might be wrong, but at least we > > > shouldn't have to wait until the aforementioned PR gets ready to be > > merged > > > to the master. > > > > > > On Wed, May 1, 2019 at 4:16 PM Justin Leet > > wrote: > > > > > > > Short version: I'm in favor of #2 of 0.7.1 and #1 as a blocker for > > 0.8.0. > > > > #3 seems like a total waste of time and effort. > > > > > > > > The wall of text version: > > > > I agree this isn't "just the wrong thing shown", but for completely > > > > different reasons. > > > > > > > > To be extremely clear about what the problem is: Our "dev" > environment > > > > (whose very name implies the audience is develops) uses a > > > performance-based > > > > advanced feature to ensure that all our of sample flows are regularly > > run > > > > and produce data. This feature has a bare minimal implementation to > be > > > > enabled via Ambari, which it currently is by default. This is because > > of > > > > the limited resources available that previously resulted in us > turning > > > off > > > > Yaf, and therefore testing it during regular full dev runs. Right now > > > > however, this feature is not exposed through the management UI, and > > > > therefore it isn't obvious what the implications are. Am I missing > > > anything > > > > here? > > > > > > > > For users actually choosing to use the parser aggregation feature in > a > > > > non-full-dev environment, I'd expect substantially more care to be > > > involved > > > > given the lack of easy configuration for it (after all, why would you > > > > bother running the aggregated parser alongside the regular parser? > This > > > > could be more explicitly stated, but again that feels like a doc
Re: [DISCUSS] Metron Release - 0.7.1 next steps
A separate [DISCUSSION] thread on Parser Aggregation support for the Management UI is coming later today. We collecting all the previous threads there which belongs to this feature and it's implementation. On Thu, May 2, 2019 at 6:32 PM Tibor Meller wrote: > I also favor option 2. I also feel it is good to highlight we are not > facing with an issue on the UI with a fix in PR#1360. > Parser aggregation had turned on for the three default parser without > having parser aggregation support added to the UI. > PR#1360 contains a whole new feature with about 6000 lines of code > changes. Which I think hold a fair amount of risk for regression. > I suggest considering this PR more than a simple patch for the parser > issue introduced in our previous release. > This is probably the biggest feature on the UI in the last one year. > Therefore am not even sure it belongs to a patch release like 0.7.x instead > of 0.8.0. > The latest version of the REST API with parser aggregation just came out > yesterday so we were able to start another round of testing. > I already identified three minor bugs. Some of them (if not all) have to > be fixed before we can consider this PR done. > Long story short: am also against the pressure to pushing this PR out ASAP. > > On Thu, May 2, 2019 at 5:50 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> As a more general question, can I ask why we're feeling pressure to push >> out a release in the first place? Again, I'm happy to continue with option >> 2. Let's move forward and get out the release. But is there a reason why >> we >> think it has to get out now, versus next week, or the week after? Otto >> pointed out a legitimate issue, dev environment or not, and I'm unclear >> why >> we have an issue with waiting for the fix. There's no pressure on this, >> imho. >> >> On Thu, May 2, 2019, 9:12 AM Otto Fowler wrote: >> >> > I remember this now, but I’m not sure how I would have related this to a >> > parser aggregation pr honestly. >> > >> > >> > On May 2, 2019 at 07:54:13, Shane Ardell (shane.m.ard...@gmail.com) >> wrote: >> > >> > Here's a link to the ngrx discussion thread from a few months back: >> > >> > >> https://lists.apache.org/thread.html/06a59ea42e8d9a9dea5f90aab4011e44434555f8b7f3cf21297c7c87@%3Cdev.metron.apache.org%3E >> > >> > On Thu, May 2, 2019 at 1:17 PM Otto Fowler >> > wrote: >> > >> > > If you can find a link in the archives for that thread, it would >> really >> > > help. >> > > >> > > I don’t think sending them up as one sensor would work…. as something >> > > quick. I think it is an interesting idea from a higher level that >> would >> > > need some more thought though ( IE: what if every sensor in the ui >> was a >> > > sensor group, and the existing entries where just groups of 1 ). >> > > >> > > As far as I can see, we have brought up the idea of a release >> ourselves, >> > I >> > > don’t see why we don’t just swarm this issue and get it right then >> > release. >> > > >> > > >> > > >> > > On May 2, 2019 at 04:16:31, Tamás Fodor (ftamas.m...@gmail.com) >> wrote: >> > > >> > > In PR#1360 we introduced a new state management strategy involving a >> new >> > > module called Ngrx. We had a discussion thread on this a few months >> ago >> > and >> > > we successfully convinced you about the benefits. This is one of the >> > > reasons why this PR is going to be still huge after cleaning up the >> > commit >> > > history. After you having a look at the changes and the feature >> itself, >> > > there's likely have questions about why certain parts work as they do. >> > The >> > > thing what I'd like to point out is that, yes, it probably takes more >> > time >> > > to get it in. >> > > >> > > In order to being able to release the RC, wouldn't it be an easy and >> > quick >> > > fix on the backend if it sent the aggregated parsers to the client as >> > they >> > > were one sensor? It's just an idea, it might be wrong, but at least we >> > > shouldn't have to wait until the aforementioned PR gets ready to be >> > merged >> > > to the master. >> > > >> > > On Wed, May 1, 2019 at 4:16 PM Justin Leet >> > wrote: >> > > >> > > > Short version: I'm in favor of #2 of 0.7.1 and #1 as a blocker for >> > 0.8.0. >> > > > #3 seems like a total waste of time and effort. >> > > > >> > > > The wall of text version: >> > > > I agree this isn't "just the wrong thing shown", but for completely >> > > > different reasons. >> > > > >> > > > To be extremely clear about what the problem is: Our "dev" >> environment >> > > > (whose very name implies the audience is develops) uses a >> > > performance-based >> > > > advanced feature to ensure that all our of sample flows are >> regularly >> > run >> > > > and produce data. This feature has a bare minimal implementation to >> be >> > > > enabled via Ambari, which it currently is by default. This is >> because >> > of >> > > > the limited resources available that previously resulted in us >> turning >> > > off >> > >
Re: [DISCUSS] Metron Release - 0.7.1 next steps
I also favor option 2. I also feel it is good to highlight we are not facing with an issue on the UI with a fix in PR#1360. Parser aggregation had turned on for the three default parser without having parser aggregation support added to the UI. PR#1360 contains a whole new feature with about 6000 lines of code changes. Which I think hold a fair amount of risk for regression. I suggest considering this PR more than a simple patch for the parser issue introduced in our previous release. This is probably the biggest feature on the UI in the last one year. Therefore am not even sure it belongs to a patch release like 0.7.x instead of 0.8.0. The latest version of the REST API with parser aggregation just came out yesterday so we were able to start another round of testing. I already identified three minor bugs. Some of them (if not all) have to be fixed before we can consider this PR done. Long story short: am also against the pressure to pushing this PR out ASAP. On Thu, May 2, 2019 at 5:50 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > As a more general question, can I ask why we're feeling pressure to push > out a release in the first place? Again, I'm happy to continue with option > 2. Let's move forward and get out the release. But is there a reason why we > think it has to get out now, versus next week, or the week after? Otto > pointed out a legitimate issue, dev environment or not, and I'm unclear why > we have an issue with waiting for the fix. There's no pressure on this, > imho. > > On Thu, May 2, 2019, 9:12 AM Otto Fowler wrote: > > > I remember this now, but I’m not sure how I would have related this to a > > parser aggregation pr honestly. > > > > > > On May 2, 2019 at 07:54:13, Shane Ardell (shane.m.ard...@gmail.com) > wrote: > > > > Here's a link to the ngrx discussion thread from a few months back: > > > > > https://lists.apache.org/thread.html/06a59ea42e8d9a9dea5f90aab4011e44434555f8b7f3cf21297c7c87@%3Cdev.metron.apache.org%3E > > > > On Thu, May 2, 2019 at 1:17 PM Otto Fowler > > wrote: > > > > > If you can find a link in the archives for that thread, it would really > > > help. > > > > > > I don’t think sending them up as one sensor would work…. as something > > > quick. I think it is an interesting idea from a higher level that would > > > need some more thought though ( IE: what if every sensor in the ui was > a > > > sensor group, and the existing entries where just groups of 1 ). > > > > > > As far as I can see, we have brought up the idea of a release > ourselves, > > I > > > don’t see why we don’t just swarm this issue and get it right then > > release. > > > > > > > > > > > > On May 2, 2019 at 04:16:31, Tamás Fodor (ftamas.m...@gmail.com) wrote: > > > > > > In PR#1360 we introduced a new state management strategy involving a > new > > > module called Ngrx. We had a discussion thread on this a few months ago > > and > > > we successfully convinced you about the benefits. This is one of the > > > reasons why this PR is going to be still huge after cleaning up the > > commit > > > history. After you having a look at the changes and the feature itself, > > > there's likely have questions about why certain parts work as they do. > > The > > > thing what I'd like to point out is that, yes, it probably takes more > > time > > > to get it in. > > > > > > In order to being able to release the RC, wouldn't it be an easy and > > quick > > > fix on the backend if it sent the aggregated parsers to the client as > > they > > > were one sensor? It's just an idea, it might be wrong, but at least we > > > shouldn't have to wait until the aforementioned PR gets ready to be > > merged > > > to the master. > > > > > > On Wed, May 1, 2019 at 4:16 PM Justin Leet > > wrote: > > > > > > > Short version: I'm in favor of #2 of 0.7.1 and #1 as a blocker for > > 0.8.0. > > > > #3 seems like a total waste of time and effort. > > > > > > > > The wall of text version: > > > > I agree this isn't "just the wrong thing shown", but for completely > > > > different reasons. > > > > > > > > To be extremely clear about what the problem is: Our "dev" > environment > > > > (whose very name implies the audience is develops) uses a > > > performance-based > > > > advanced feature to ensure that all our of sample flows are regularly > > run > > > > and produce data. This feature has a bare minimal implementation to > be > > > > enabled via Ambari, which it currently is by default. This is because > > of > > > > the limited resources available that previously resulted in us > turning > > > off > > > > Yaf, and therefore testing it during regular full dev runs. Right now > > > > however, this feature is not exposed through the management UI, and > > > > therefore it isn't obvious what the implications are. Am I missing > > > anything > > > > here? > > > > > > > > For users actually choosing to use the parser aggregation feature in > a > > > > non-full-dev environment, I'd expect substantially more care to be
Re: [DISCUSS] Metron Release - 0.7.1 next steps
As a more general question, can I ask why we're feeling pressure to push out a release in the first place? Again, I'm happy to continue with option 2. Let's move forward and get out the release. But is there a reason why we think it has to get out now, versus next week, or the week after? Otto pointed out a legitimate issue, dev environment or not, and I'm unclear why we have an issue with waiting for the fix. There's no pressure on this, imho. On Thu, May 2, 2019, 9:12 AM Otto Fowler wrote: > I remember this now, but I’m not sure how I would have related this to a > parser aggregation pr honestly. > > > On May 2, 2019 at 07:54:13, Shane Ardell (shane.m.ard...@gmail.com) wrote: > > Here's a link to the ngrx discussion thread from a few months back: > > https://lists.apache.org/thread.html/06a59ea42e8d9a9dea5f90aab4011e44434555f8b7f3cf21297c7c87@%3Cdev.metron.apache.org%3E > > On Thu, May 2, 2019 at 1:17 PM Otto Fowler > wrote: > > > If you can find a link in the archives for that thread, it would really > > help. > > > > I don’t think sending them up as one sensor would work…. as something > > quick. I think it is an interesting idea from a higher level that would > > need some more thought though ( IE: what if every sensor in the ui was a > > sensor group, and the existing entries where just groups of 1 ). > > > > As far as I can see, we have brought up the idea of a release ourselves, > I > > don’t see why we don’t just swarm this issue and get it right then > release. > > > > > > > > On May 2, 2019 at 04:16:31, Tamás Fodor (ftamas.m...@gmail.com) wrote: > > > > In PR#1360 we introduced a new state management strategy involving a new > > module called Ngrx. We had a discussion thread on this a few months ago > and > > we successfully convinced you about the benefits. This is one of the > > reasons why this PR is going to be still huge after cleaning up the > commit > > history. After you having a look at the changes and the feature itself, > > there's likely have questions about why certain parts work as they do. > The > > thing what I'd like to point out is that, yes, it probably takes more > time > > to get it in. > > > > In order to being able to release the RC, wouldn't it be an easy and > quick > > fix on the backend if it sent the aggregated parsers to the client as > they > > were one sensor? It's just an idea, it might be wrong, but at least we > > shouldn't have to wait until the aforementioned PR gets ready to be > merged > > to the master. > > > > On Wed, May 1, 2019 at 4:16 PM Justin Leet > wrote: > > > > > Short version: I'm in favor of #2 of 0.7.1 and #1 as a blocker for > 0.8.0. > > > #3 seems like a total waste of time and effort. > > > > > > The wall of text version: > > > I agree this isn't "just the wrong thing shown", but for completely > > > different reasons. > > > > > > To be extremely clear about what the problem is: Our "dev" environment > > > (whose very name implies the audience is develops) uses a > > performance-based > > > advanced feature to ensure that all our of sample flows are regularly > run > > > and produce data. This feature has a bare minimal implementation to be > > > enabled via Ambari, which it currently is by default. This is because > of > > > the limited resources available that previously resulted in us turning > > off > > > Yaf, and therefore testing it during regular full dev runs. Right now > > > however, this feature is not exposed through the management UI, and > > > therefore it isn't obvious what the implications are. Am I missing > > anything > > > here? > > > > > > For users actually choosing to use the parser aggregation feature in a > > > non-full-dev environment, I'd expect substantially more care to be > > involved > > > given the lack of easy configuration for it (after all, why would you > > > bother running the aggregated parser alongside the regular parser? This > > > could be more explicitly stated, but again that feels like a doc > problem. > > > Right now I could essentially provide two of the same parser and create > > the > > > same problem, so right now aggregation is only special because it runs > on > > > dev by default). This is, in my opinion, primarily a first impression > > > problem and likely one of many areas that could use improved > > documentation. > > > > > > Quite frankly, I think the issue pointed out here could mostly be > > resolved > > > by documenting how the current aggregation is done in dev, and telling > > how > > > to change it. Especially for a 0.x.1 release, which is primarily bug > > > fixes. As can be inferred from my vote, I don't think this problem is a > > > problem that needs solving in a point release. I would support > improving > > > the documentation, both full-dev and for aggregation in general for the > > > 0.7.1 point release, while making a 0.8.0 release contingent upon the > > > outstanding PRs to enable it in the management UI. > > > > > > There are a couple deeper issues, imo, that I
Re: [DISCUSS] Metron Release - 0.7.1 next steps
I remember this now, but I’m not sure how I would have related this to a parser aggregation pr honestly. On May 2, 2019 at 07:54:13, Shane Ardell (shane.m.ard...@gmail.com) wrote: Here's a link to the ngrx discussion thread from a few months back: https://lists.apache.org/thread.html/06a59ea42e8d9a9dea5f90aab4011e44434555f8b7f3cf21297c7c87@%3Cdev.metron.apache.org%3E On Thu, May 2, 2019 at 1:17 PM Otto Fowler wrote: > If you can find a link in the archives for that thread, it would really > help. > > I don’t think sending them up as one sensor would work…. as something > quick. I think it is an interesting idea from a higher level that would > need some more thought though ( IE: what if every sensor in the ui was a > sensor group, and the existing entries where just groups of 1 ). > > As far as I can see, we have brought up the idea of a release ourselves, I > don’t see why we don’t just swarm this issue and get it right then release. > > > > On May 2, 2019 at 04:16:31, Tamás Fodor (ftamas.m...@gmail.com) wrote: > > In PR#1360 we introduced a new state management strategy involving a new > module called Ngrx. We had a discussion thread on this a few months ago and > we successfully convinced you about the benefits. This is one of the > reasons why this PR is going to be still huge after cleaning up the commit > history. After you having a look at the changes and the feature itself, > there's likely have questions about why certain parts work as they do. The > thing what I'd like to point out is that, yes, it probably takes more time > to get it in. > > In order to being able to release the RC, wouldn't it be an easy and quick > fix on the backend if it sent the aggregated parsers to the client as they > were one sensor? It's just an idea, it might be wrong, but at least we > shouldn't have to wait until the aforementioned PR gets ready to be merged > to the master. > > On Wed, May 1, 2019 at 4:16 PM Justin Leet wrote: > > > Short version: I'm in favor of #2 of 0.7.1 and #1 as a blocker for 0.8.0. > > #3 seems like a total waste of time and effort. > > > > The wall of text version: > > I agree this isn't "just the wrong thing shown", but for completely > > different reasons. > > > > To be extremely clear about what the problem is: Our "dev" environment > > (whose very name implies the audience is develops) uses a > performance-based > > advanced feature to ensure that all our of sample flows are regularly run > > and produce data. This feature has a bare minimal implementation to be > > enabled via Ambari, which it currently is by default. This is because of > > the limited resources available that previously resulted in us turning > off > > Yaf, and therefore testing it during regular full dev runs. Right now > > however, this feature is not exposed through the management UI, and > > therefore it isn't obvious what the implications are. Am I missing > anything > > here? > > > > For users actually choosing to use the parser aggregation feature in a > > non-full-dev environment, I'd expect substantially more care to be > involved > > given the lack of easy configuration for it (after all, why would you > > bother running the aggregated parser alongside the regular parser? This > > could be more explicitly stated, but again that feels like a doc problem. > > Right now I could essentially provide two of the same parser and create > the > > same problem, so right now aggregation is only special because it runs on > > dev by default). This is, in my opinion, primarily a first impression > > problem and likely one of many areas that could use improved > documentation. > > > > Quite frankly, I think the issue pointed out here could mostly be > resolved > > by documenting how the current aggregation is done in dev, and telling > how > > to change it. Especially for a 0.x.1 release, which is primarily bug > > fixes. As can be inferred from my vote, I don't think this problem is a > > problem that needs solving in a point release. I would support improving > > the documentation, both full-dev and for aggregation in general for the > > 0.7.1 point release, while making a 0.8.0 release contingent upon the > > outstanding PRs to enable it in the management UI. > > > > There are a couple deeper issues, imo, that I care substantially more > about > > than this in particular > > * The dev environment is being used as our intro for users, because it's > > convenient for us to not maintain more environments (which has been a > major > > pain point in the past). Worse, the dev environment strongly implies it's > > for Metron developers, rather than people looking to build on top of > > Metron. We need an actual strategy for providing end users a clean > > impression of Metron (this could be clarifying what the expectations of > > full dev are, renaming it to something like "full-demo", something more > > involved, etc.). This is something that we've needed for awhile in > general, > > and includes larger topics like
Re: [DISCUSS] Metron Release - 0.7.1 next steps
FWIW, I'm in favor of 2. I think it's a relatively minor bug and the impact is limited. I do agree that it should be a blocker for 0.8.0 though. On Thu, May 2, 2019 at 9:31 AM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > I am still in favor of option 2. I will volunteer and submit the doc PR. I > agree we should not rush through a review process for a maintenance > release. The implications to the UI, as Otto asked, are that aggregated > parsers will not show up in the UI. You cannot create them there. Actually, > any parser not created through the UI (eg CLI) will not show up in the UI, > aggregated or not. > > As a separate issue, I will also volunteer to see if I can help Tamas find > the discuss thread mentioned. It should be linked to the PR or feature > branch for reference. That may also be a gap in dev guidelines that should > be spelled out. > > On Thu, May 2, 2019, 7:17 AM Nick Allen wrote: > > > To echo Justin's comments, I am in favor of #2, which provides a clear, > > well-defined path to a release. > > > >- Why hold back a release, especially a point release containing 89 > >improvements, for one issue that will not affect most users? > > > > > >- It is one thing to stall a release to address a bug of limited > scope, > >where a fix is well understood and ready for review, but it is > > completely > >another issue to delay for this. > > > > > >- I don't see a set of reviewable PRs yet that will push this over the > >finish line. As has been noted, there were fundamental problems with > > #1360 > >(which has now been closed) that would have prevented adequate review > by > >the community. > > > > > >- Why drive this issue with the pressure of a stalled release, instead > >of just releasing the fix when it is ready and has been adequately > >reviewed? Swarming on an issue does not often produce quality > results. > > > > For those in favor of #1, can someone please provide a clear outline of > > what the fix looks-like? How many PRs will this require? When are these > > PRs likely to be ready? Who is driving this? Tamás has already > commented > > that this not a quick fix. This path is very murky to me, but maybe I am > > just ignorant on this. > > > > I would also urge other committers and users who don't have a binding > vote > > on the release to share their opinion on the path forward. > > > > > > > > > > On Thu, May 2, 2019 at 7:17 AM Otto Fowler > > wrote: > > > > > If you can find a link in the archives for that thread, it would really > > > help. > > > > > > I don’t think sending them up as one sensor would work…. as something > > > quick. I think it is an interesting idea from a higher level that > would > > > need some more thought though ( IE: what if every sensor in the ui was > a > > > sensor group, and the existing entries where just groups of 1 ). > > > > > > As far as I can see, we have brought up the idea of a release > ourselves, > > I > > > don’t see why we don’t just swarm this issue and get it right then > > release. > > > > > > > > > > > > On May 2, 2019 at 04:16:31, Tamás Fodor (ftamas.m...@gmail.com) wrote: > > > > > > In PR#1360 we introduced a new state management strategy involving a > new > > > module called Ngrx. We had a discussion thread on this a few months ago > > and > > > we successfully convinced you about the benefits. This is one of the > > > reasons why this PR is going to be still huge after cleaning up the > > commit > > > history. After you having a look at the changes and the feature itself, > > > there's likely have questions about why certain parts work as they do. > > The > > > thing what I'd like to point out is that, yes, it probably takes more > > time > > > to get it in. > > > > > > In order to being able to release the RC, wouldn't it be an easy and > > quick > > > fix on the backend if it sent the aggregated parsers to the client as > > they > > > were one sensor? It's just an idea, it might be wrong, but at least we > > > shouldn't have to wait until the aforementioned PR gets ready to be > > merged > > > to the master. > > > > > > On Wed, May 1, 2019 at 4:16 PM Justin Leet > > wrote: > > > > > > > Short version: I'm in favor of #2 of 0.7.1 and #1 as a blocker for > > 0.8.0. > > > > #3 seems like a total waste of time and effort. > > > > > > > > The wall of text version: > > > > I agree this isn't "just the wrong thing shown", but for completely > > > > different reasons. > > > > > > > > To be extremely clear about what the problem is: Our "dev" > environment > > > > (whose very name implies the audience is develops) uses a > > > performance-based > > > > advanced feature to ensure that all our of sample flows are regularly > > run > > > > and produce data. This feature has a bare minimal implementation to > be > > > > enabled via Ambari, which it currently is by default. This is because > > of > > > > the limited resources available that previously resulted in us >
Re: [DISCUSS] Metron Release - 0.7.1 next steps
I am still in favor of option 2. I will volunteer and submit the doc PR. I agree we should not rush through a review process for a maintenance release. The implications to the UI, as Otto asked, are that aggregated parsers will not show up in the UI. You cannot create them there. Actually, any parser not created through the UI (eg CLI) will not show up in the UI, aggregated or not. As a separate issue, I will also volunteer to see if I can help Tamas find the discuss thread mentioned. It should be linked to the PR or feature branch for reference. That may also be a gap in dev guidelines that should be spelled out. On Thu, May 2, 2019, 7:17 AM Nick Allen wrote: > To echo Justin's comments, I am in favor of #2, which provides a clear, > well-defined path to a release. > >- Why hold back a release, especially a point release containing 89 >improvements, for one issue that will not affect most users? > > >- It is one thing to stall a release to address a bug of limited scope, >where a fix is well understood and ready for review, but it is > completely >another issue to delay for this. > > >- I don't see a set of reviewable PRs yet that will push this over the >finish line. As has been noted, there were fundamental problems with > #1360 >(which has now been closed) that would have prevented adequate review by >the community. > > >- Why drive this issue with the pressure of a stalled release, instead >of just releasing the fix when it is ready and has been adequately >reviewed? Swarming on an issue does not often produce quality results. > > For those in favor of #1, can someone please provide a clear outline of > what the fix looks-like? How many PRs will this require? When are these > PRs likely to be ready? Who is driving this? Tamás has already commented > that this not a quick fix. This path is very murky to me, but maybe I am > just ignorant on this. > > I would also urge other committers and users who don't have a binding vote > on the release to share their opinion on the path forward. > > > > > On Thu, May 2, 2019 at 7:17 AM Otto Fowler > wrote: > > > If you can find a link in the archives for that thread, it would really > > help. > > > > I don’t think sending them up as one sensor would work…. as something > > quick. I think it is an interesting idea from a higher level that would > > need some more thought though ( IE: what if every sensor in the ui was a > > sensor group, and the existing entries where just groups of 1 ). > > > > As far as I can see, we have brought up the idea of a release ourselves, > I > > don’t see why we don’t just swarm this issue and get it right then > release. > > > > > > > > On May 2, 2019 at 04:16:31, Tamás Fodor (ftamas.m...@gmail.com) wrote: > > > > In PR#1360 we introduced a new state management strategy involving a new > > module called Ngrx. We had a discussion thread on this a few months ago > and > > we successfully convinced you about the benefits. This is one of the > > reasons why this PR is going to be still huge after cleaning up the > commit > > history. After you having a look at the changes and the feature itself, > > there's likely have questions about why certain parts work as they do. > The > > thing what I'd like to point out is that, yes, it probably takes more > time > > to get it in. > > > > In order to being able to release the RC, wouldn't it be an easy and > quick > > fix on the backend if it sent the aggregated parsers to the client as > they > > were one sensor? It's just an idea, it might be wrong, but at least we > > shouldn't have to wait until the aforementioned PR gets ready to be > merged > > to the master. > > > > On Wed, May 1, 2019 at 4:16 PM Justin Leet > wrote: > > > > > Short version: I'm in favor of #2 of 0.7.1 and #1 as a blocker for > 0.8.0. > > > #3 seems like a total waste of time and effort. > > > > > > The wall of text version: > > > I agree this isn't "just the wrong thing shown", but for completely > > > different reasons. > > > > > > To be extremely clear about what the problem is: Our "dev" environment > > > (whose very name implies the audience is develops) uses a > > performance-based > > > advanced feature to ensure that all our of sample flows are regularly > run > > > and produce data. This feature has a bare minimal implementation to be > > > enabled via Ambari, which it currently is by default. This is because > of > > > the limited resources available that previously resulted in us turning > > off > > > Yaf, and therefore testing it during regular full dev runs. Right now > > > however, this feature is not exposed through the management UI, and > > > therefore it isn't obvious what the implications are. Am I missing > > anything > > > here? > > > > > > For users actually choosing to use the parser aggregation feature in a > > > non-full-dev environment, I'd expect substantially more care to be > > involved > > > given the lack of easy configuration for
Re: [DISCUSS] Metron Release - 0.7.1 next steps
To echo Justin's comments, I am in favor of #2, which provides a clear, well-defined path to a release. - Why hold back a release, especially a point release containing 89 improvements, for one issue that will not affect most users? - It is one thing to stall a release to address a bug of limited scope, where a fix is well understood and ready for review, but it is completely another issue to delay for this. - I don't see a set of reviewable PRs yet that will push this over the finish line. As has been noted, there were fundamental problems with #1360 (which has now been closed) that would have prevented adequate review by the community. - Why drive this issue with the pressure of a stalled release, instead of just releasing the fix when it is ready and has been adequately reviewed? Swarming on an issue does not often produce quality results. For those in favor of #1, can someone please provide a clear outline of what the fix looks-like? How many PRs will this require? When are these PRs likely to be ready? Who is driving this? Tamás has already commented that this not a quick fix. This path is very murky to me, but maybe I am just ignorant on this. I would also urge other committers and users who don't have a binding vote on the release to share their opinion on the path forward. On Thu, May 2, 2019 at 7:17 AM Otto Fowler wrote: > If you can find a link in the archives for that thread, it would really > help. > > I don’t think sending them up as one sensor would work…. as something > quick. I think it is an interesting idea from a higher level that would > need some more thought though ( IE: what if every sensor in the ui was a > sensor group, and the existing entries where just groups of 1 ). > > As far as I can see, we have brought up the idea of a release ourselves, I > don’t see why we don’t just swarm this issue and get it right then release. > > > > On May 2, 2019 at 04:16:31, Tamás Fodor (ftamas.m...@gmail.com) wrote: > > In PR#1360 we introduced a new state management strategy involving a new > module called Ngrx. We had a discussion thread on this a few months ago and > we successfully convinced you about the benefits. This is one of the > reasons why this PR is going to be still huge after cleaning up the commit > history. After you having a look at the changes and the feature itself, > there's likely have questions about why certain parts work as they do. The > thing what I'd like to point out is that, yes, it probably takes more time > to get it in. > > In order to being able to release the RC, wouldn't it be an easy and quick > fix on the backend if it sent the aggregated parsers to the client as they > were one sensor? It's just an idea, it might be wrong, but at least we > shouldn't have to wait until the aforementioned PR gets ready to be merged > to the master. > > On Wed, May 1, 2019 at 4:16 PM Justin Leet wrote: > > > Short version: I'm in favor of #2 of 0.7.1 and #1 as a blocker for 0.8.0. > > #3 seems like a total waste of time and effort. > > > > The wall of text version: > > I agree this isn't "just the wrong thing shown", but for completely > > different reasons. > > > > To be extremely clear about what the problem is: Our "dev" environment > > (whose very name implies the audience is develops) uses a > performance-based > > advanced feature to ensure that all our of sample flows are regularly run > > and produce data. This feature has a bare minimal implementation to be > > enabled via Ambari, which it currently is by default. This is because of > > the limited resources available that previously resulted in us turning > off > > Yaf, and therefore testing it during regular full dev runs. Right now > > however, this feature is not exposed through the management UI, and > > therefore it isn't obvious what the implications are. Am I missing > anything > > here? > > > > For users actually choosing to use the parser aggregation feature in a > > non-full-dev environment, I'd expect substantially more care to be > involved > > given the lack of easy configuration for it (after all, why would you > > bother running the aggregated parser alongside the regular parser? This > > could be more explicitly stated, but again that feels like a doc problem. > > Right now I could essentially provide two of the same parser and create > the > > same problem, so right now aggregation is only special because it runs on > > dev by default). This is, in my opinion, primarily a first impression > > problem and likely one of many areas that could use improved > documentation. > > > > Quite frankly, I think the issue pointed out here could mostly be > resolved > > by documenting how the current aggregation is done in dev, and telling > how > > to change it. Especially for a 0.x.1 release, which is primarily bug > > fixes. As can be inferred from my vote, I don't think this problem is a > > problem that needs solving in a point release. I would
Re: [DISCUSS] Metron Release - 0.7.1 next steps
Here's a link to the ngrx discussion thread from a few months back: https://lists.apache.org/thread.html/06a59ea42e8d9a9dea5f90aab4011e44434555f8b7f3cf21297c7c87@%3Cdev.metron.apache.org%3E On Thu, May 2, 2019 at 1:17 PM Otto Fowler wrote: > If you can find a link in the archives for that thread, it would really > help. > > I don’t think sending them up as one sensor would work…. as something > quick. I think it is an interesting idea from a higher level that would > need some more thought though ( IE: what if every sensor in the ui was a > sensor group, and the existing entries where just groups of 1 ). > > As far as I can see, we have brought up the idea of a release ourselves, I > don’t see why we don’t just swarm this issue and get it right then release. > > > > On May 2, 2019 at 04:16:31, Tamás Fodor (ftamas.m...@gmail.com) wrote: > > In PR#1360 we introduced a new state management strategy involving a new > module called Ngrx. We had a discussion thread on this a few months ago and > we successfully convinced you about the benefits. This is one of the > reasons why this PR is going to be still huge after cleaning up the commit > history. After you having a look at the changes and the feature itself, > there's likely have questions about why certain parts work as they do. The > thing what I'd like to point out is that, yes, it probably takes more time > to get it in. > > In order to being able to release the RC, wouldn't it be an easy and quick > fix on the backend if it sent the aggregated parsers to the client as they > were one sensor? It's just an idea, it might be wrong, but at least we > shouldn't have to wait until the aforementioned PR gets ready to be merged > to the master. > > On Wed, May 1, 2019 at 4:16 PM Justin Leet wrote: > > > Short version: I'm in favor of #2 of 0.7.1 and #1 as a blocker for 0.8.0. > > #3 seems like a total waste of time and effort. > > > > The wall of text version: > > I agree this isn't "just the wrong thing shown", but for completely > > different reasons. > > > > To be extremely clear about what the problem is: Our "dev" environment > > (whose very name implies the audience is develops) uses a > performance-based > > advanced feature to ensure that all our of sample flows are regularly run > > and produce data. This feature has a bare minimal implementation to be > > enabled via Ambari, which it currently is by default. This is because of > > the limited resources available that previously resulted in us turning > off > > Yaf, and therefore testing it during regular full dev runs. Right now > > however, this feature is not exposed through the management UI, and > > therefore it isn't obvious what the implications are. Am I missing > anything > > here? > > > > For users actually choosing to use the parser aggregation feature in a > > non-full-dev environment, I'd expect substantially more care to be > involved > > given the lack of easy configuration for it (after all, why would you > > bother running the aggregated parser alongside the regular parser? This > > could be more explicitly stated, but again that feels like a doc problem. > > Right now I could essentially provide two of the same parser and create > the > > same problem, so right now aggregation is only special because it runs on > > dev by default). This is, in my opinion, primarily a first impression > > problem and likely one of many areas that could use improved > documentation. > > > > Quite frankly, I think the issue pointed out here could mostly be > resolved > > by documenting how the current aggregation is done in dev, and telling > how > > to change it. Especially for a 0.x.1 release, which is primarily bug > > fixes. As can be inferred from my vote, I don't think this problem is a > > problem that needs solving in a point release. I would support improving > > the documentation, both full-dev and for aggregation in general for the > > 0.7.1 point release, while making a 0.8.0 release contingent upon the > > outstanding PRs to enable it in the management UI. > > > > There are a couple deeper issues, imo, that I care substantially more > about > > than this in particular > > * The dev environment is being used as our intro for users, because it's > > convenient for us to not maintain more environments (which has been a > major > > pain point in the past). Worse, the dev environment strongly implies it's > > for Metron developers, rather than people looking to build on top of > > Metron. We need an actual strategy for providing end users a clean > > impression of Metron (this could be clarifying what the expectations of > > full dev are, renaming it to something like "full-demo", something more > > involved, etc.). This is something that we've needed for awhile in > general, > > and includes larger topics like improving our website, potentially > > improving the site book, actually publishing our Javadocs somewhere so > > people can develop things easier, publishing out info about Stellar
Re: [DISCUSS] Metron Release - 0.7.1 next steps
If you can find a link in the archives for that thread, it would really help. I don’t think sending them up as one sensor would work…. as something quick. I think it is an interesting idea from a higher level that would need some more thought though ( IE: what if every sensor in the ui was a sensor group, and the existing entries where just groups of 1 ). As far as I can see, we have brought up the idea of a release ourselves, I don’t see why we don’t just swarm this issue and get it right then release. On May 2, 2019 at 04:16:31, Tamás Fodor (ftamas.m...@gmail.com) wrote: In PR#1360 we introduced a new state management strategy involving a new module called Ngrx. We had a discussion thread on this a few months ago and we successfully convinced you about the benefits. This is one of the reasons why this PR is going to be still huge after cleaning up the commit history. After you having a look at the changes and the feature itself, there's likely have questions about why certain parts work as they do. The thing what I'd like to point out is that, yes, it probably takes more time to get it in. In order to being able to release the RC, wouldn't it be an easy and quick fix on the backend if it sent the aggregated parsers to the client as they were one sensor? It's just an idea, it might be wrong, but at least we shouldn't have to wait until the aforementioned PR gets ready to be merged to the master. On Wed, May 1, 2019 at 4:16 PM Justin Leet wrote: > Short version: I'm in favor of #2 of 0.7.1 and #1 as a blocker for 0.8.0. > #3 seems like a total waste of time and effort. > > The wall of text version: > I agree this isn't "just the wrong thing shown", but for completely > different reasons. > > To be extremely clear about what the problem is: Our "dev" environment > (whose very name implies the audience is develops) uses a performance-based > advanced feature to ensure that all our of sample flows are regularly run > and produce data. This feature has a bare minimal implementation to be > enabled via Ambari, which it currently is by default. This is because of > the limited resources available that previously resulted in us turning off > Yaf, and therefore testing it during regular full dev runs. Right now > however, this feature is not exposed through the management UI, and > therefore it isn't obvious what the implications are. Am I missing anything > here? > > For users actually choosing to use the parser aggregation feature in a > non-full-dev environment, I'd expect substantially more care to be involved > given the lack of easy configuration for it (after all, why would you > bother running the aggregated parser alongside the regular parser? This > could be more explicitly stated, but again that feels like a doc problem. > Right now I could essentially provide two of the same parser and create the > same problem, so right now aggregation is only special because it runs on > dev by default). This is, in my opinion, primarily a first impression > problem and likely one of many areas that could use improved documentation. > > Quite frankly, I think the issue pointed out here could mostly be resolved > by documenting how the current aggregation is done in dev, and telling how > to change it. Especially for a 0.x.1 release, which is primarily bug > fixes. As can be inferred from my vote, I don't think this problem is a > problem that needs solving in a point release. I would support improving > the documentation, both full-dev and for aggregation in general for the > 0.7.1 point release, while making a 0.8.0 release contingent upon the > outstanding PRs to enable it in the management UI. > > There are a couple deeper issues, imo, that I care substantially more about > than this in particular > * The dev environment is being used as our intro for users, because it's > convenient for us to not maintain more environments (which has been a major > pain point in the past). Worse, the dev environment strongly implies it's > for Metron developers, rather than people looking to build on top of > Metron. We need an actual strategy for providing end users a clean > impression of Metron (this could be clarifying what the expectations of > full dev are, renaming it to something like "full-demo", something more > involved, etc.). This is something that we've needed for awhile in general, > and includes larger topics like improving our website, potentially > improving the site book, actually publishing our Javadocs somewhere so > people can develop things easier, publishing out info about Stellar > functions in a better manner, etc. > * The fact that parsers are handled in Ambari at all. It's awful and leads > to situations like this. To the best of my knowledge, once we can do > chaining and aggregation in the Management UI, we should be able to > entirely divorce these two overlapping domains. I'd love to see parsers > ripped out of Ambari, then full-dev manages all the setup via REST. At that > point, we
Re: [DISCUSS] Metron Release - 0.7.1 next steps
In PR#1360 we introduced a new state management strategy involving a new module called Ngrx. We had a discussion thread on this a few months ago and we successfully convinced you about the benefits. This is one of the reasons why this PR is going to be still huge after cleaning up the commit history. After you having a look at the changes and the feature itself, there's likely have questions about why certain parts work as they do. The thing what I'd like to point out is that, yes, it probably takes more time to get it in. In order to being able to release the RC, wouldn't it be an easy and quick fix on the backend if it sent the aggregated parsers to the client as they were one sensor? It's just an idea, it might be wrong, but at least we shouldn't have to wait until the aforementioned PR gets ready to be merged to the master. On Wed, May 1, 2019 at 4:16 PM Justin Leet wrote: > Short version: I'm in favor of #2 of 0.7.1 and #1 as a blocker for 0.8.0. > #3 seems like a total waste of time and effort. > > The wall of text version: > I agree this isn't "just the wrong thing shown", but for completely > different reasons. > > To be extremely clear about what the problem is: Our "dev" environment > (whose very name implies the audience is develops) uses a performance-based > advanced feature to ensure that all our of sample flows are regularly run > and produce data. This feature has a bare minimal implementation to be > enabled via Ambari, which it currently is by default. This is because of > the limited resources available that previously resulted in us turning off > Yaf, and therefore testing it during regular full dev runs. Right now > however, this feature is not exposed through the management UI, and > therefore it isn't obvious what the implications are. Am I missing anything > here? > > For users actually choosing to use the parser aggregation feature in a > non-full-dev environment, I'd expect substantially more care to be involved > given the lack of easy configuration for it (after all, why would you > bother running the aggregated parser alongside the regular parser? This > could be more explicitly stated, but again that feels like a doc problem. > Right now I could essentially provide two of the same parser and create the > same problem, so right now aggregation is only special because it runs on > dev by default). This is, in my opinion, primarily a first impression > problem and likely one of many areas that could use improved documentation. > > Quite frankly, I think the issue pointed out here could mostly be resolved > by documenting how the current aggregation is done in dev, and telling how > to change it. Especially for a 0.x.1 release, which is primarily bug > fixes. As can be inferred from my vote, I don't think this problem is a > problem that needs solving in a point release. I would support improving > the documentation, both full-dev and for aggregation in general for the > 0.7.1 point release, while making a 0.8.0 release contingent upon the > outstanding PRs to enable it in the management UI. > > There are a couple deeper issues, imo, that I care substantially more about > than this in particular > * The dev environment is being used as our intro for users, because it's > convenient for us to not maintain more environments (which has been a major > pain point in the past). Worse, the dev environment strongly implies it's > for Metron developers, rather than people looking to build on top of > Metron. We need an actual strategy for providing end users a clean > impression of Metron (this could be clarifying what the expectations of > full dev are, renaming it to something like "full-demo", something more > involved, etc.). This is something that we've needed for awhile in general, > and includes larger topics like improving our website, potentially > improving the site book, actually publishing our Javadocs somewhere so > people can develop things easier, publishing out info about Stellar > functions in a better manner, etc. > * The fact that parsers are handled in Ambari at all. It's awful and leads > to situations like this. To the best of my knowledge, once we can do > chaining and aggregation in the Management UI, we should be able to > entirely divorce these two overlapping domains. I'd love to see parsers > ripped out of Ambari, then full-dev manages all the setup via REST. At that > point, we can easily tell everyone to just use the management UI. > > On Wed, May 1, 2019 at 7:23 AM Otto Fowler > wrote: > > > I think it would help if the full consequences of having the UI show the > > wrong status where listed. > > > > Someone trying metron, will, by default , see the wrong thing in the UI > for > > the ONLY sensors they have that are running and doing data. > > > > What happens when they try to start them to make them work? One, two or > > all? > > What happens when he edits them or try to add transformations? One, two > or > > all? > > What other things can you do
Re: [DISCUSS] Metron Release - 0.7.1 next steps
Short version: I'm in favor of #2 of 0.7.1 and #1 as a blocker for 0.8.0. #3 seems like a total waste of time and effort. The wall of text version: I agree this isn't "just the wrong thing shown", but for completely different reasons. To be extremely clear about what the problem is: Our "dev" environment (whose very name implies the audience is develops) uses a performance-based advanced feature to ensure that all our of sample flows are regularly run and produce data. This feature has a bare minimal implementation to be enabled via Ambari, which it currently is by default. This is because of the limited resources available that previously resulted in us turning off Yaf, and therefore testing it during regular full dev runs. Right now however, this feature is not exposed through the management UI, and therefore it isn't obvious what the implications are. Am I missing anything here? For users actually choosing to use the parser aggregation feature in a non-full-dev environment, I'd expect substantially more care to be involved given the lack of easy configuration for it (after all, why would you bother running the aggregated parser alongside the regular parser? This could be more explicitly stated, but again that feels like a doc problem. Right now I could essentially provide two of the same parser and create the same problem, so right now aggregation is only special because it runs on dev by default). This is, in my opinion, primarily a first impression problem and likely one of many areas that could use improved documentation. Quite frankly, I think the issue pointed out here could mostly be resolved by documenting how the current aggregation is done in dev, and telling how to change it. Especially for a 0.x.1 release, which is primarily bug fixes. As can be inferred from my vote, I don't think this problem is a problem that needs solving in a point release. I would support improving the documentation, both full-dev and for aggregation in general for the 0.7.1 point release, while making a 0.8.0 release contingent upon the outstanding PRs to enable it in the management UI. There are a couple deeper issues, imo, that I care substantially more about than this in particular * The dev environment is being used as our intro for users, because it's convenient for us to not maintain more environments (which has been a major pain point in the past). Worse, the dev environment strongly implies it's for Metron developers, rather than people looking to build on top of Metron. We need an actual strategy for providing end users a clean impression of Metron (this could be clarifying what the expectations of full dev are, renaming it to something like "full-demo", something more involved, etc.). This is something that we've needed for awhile in general, and includes larger topics like improving our website, potentially improving the site book, actually publishing our Javadocs somewhere so people can develop things easier, publishing out info about Stellar functions in a better manner, etc. * The fact that parsers are handled in Ambari at all. It's awful and leads to situations like this. To the best of my knowledge, once we can do chaining and aggregation in the Management UI, we should be able to entirely divorce these two overlapping domains. I'd love to see parsers ripped out of Ambari, then full-dev manages all the setup via REST. At that point, we can easily tell everyone to just use the management UI. On Wed, May 1, 2019 at 7:23 AM Otto Fowler wrote: > I think it would help if the full consequences of having the UI show the > wrong status where listed. > > Someone trying metron, will, by default , see the wrong thing in the UI for > the ONLY sensors they have that are running and doing data. > > What happens when they try to start them to make them work? One, two or > all? > What happens when he edits them or try to add transformations? One, two or > all? > What other things can you do with the sensors in the ui? What happens? > > Are we recommending aggregation on the list and elsewhere for users? Are > we recommending something that is going to ensure they get into this > situation? > > I think this is more than ‘just the wrong thing shown’ in the ui. > > > > > On April 30, 2019 at 20:48:10, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > The vote for RC1 did not pass and I'd like to kickstart some discussion > about what we should do. > > I started taking a look at PR#1360 and it looks like this isn't quite as > close to being able go in as I had originally expected. I want to talk > about options here. It seems to me that we can: > > 1. Wait for PR#1360 to go in, but this is likely going to take more time > than originally anticipated > 2. Accept the issue in full dev, but add some notes in the developer > docs about the current feature gap and why sensors aren't showing status in > the management UI when aggregation is enabled. > 3. Find some other workable UI solution. > 4.
Re: [DISCUSS] Metron Release - 0.7.1 next steps
I think it would help if the full consequences of having the UI show the wrong status where listed. Someone trying metron, will, by default , see the wrong thing in the UI for the ONLY sensors they have that are running and doing data. What happens when they try to start them to make them work? One, two or all? What happens when he edits them or try to add transformations? One, two or all? What other things can you do with the sensors in the ui? What happens? Are we recommending aggregation on the list and elsewhere for users? Are we recommending something that is going to ensure they get into this situation? I think this is more than ‘just the wrong thing shown’ in the ui. On April 30, 2019 at 20:48:10, Michael Miklavcic ( michael.miklav...@gmail.com) wrote: The vote for RC1 did not pass and I'd like to kickstart some discussion about what we should do. I started taking a look at PR#1360 and it looks like this isn't quite as close to being able go in as I had originally expected. I want to talk about options here. It seems to me that we can: 1. Wait for PR#1360 to go in, but this is likely going to take more time than originally anticipated 2. Accept the issue in full dev, but add some notes in the developer docs about the current feature gap and why sensors aren't showing status in the management UI when aggregation is enabled. 3. Find some other workable UI solution. 4. Other option? All things considered, I'm personally leaning towards #2 in the short-term, but I think we should probably talk about this a bit before deciding what RC2 should be. Best, Mike
[DISCUSS] Metron Release - 0.7.1 next steps
The vote for RC1 did not pass and I'd like to kickstart some discussion about what we should do. I started taking a look at PR#1360 and it looks like this isn't quite as close to being able go in as I had originally expected. I want to talk about options here. It seems to me that we can: 1. Wait for PR#1360 to go in, but this is likely going to take more time than originally anticipated 2. Accept the issue in full dev, but add some notes in the developer docs about the current feature gap and why sensors aren't showing status in the management UI when aggregation is enabled. 3. Find some other workable UI solution. 4. Other option? All things considered, I'm personally leaning towards #2 in the short-term, but I think we should probably talk about this a bit before deciding what RC2 should be. Best, Mike
