Here's the ticket I created to track it, which also references the Jira for the new UI feature. https://issues.apache.org/jira/browse/METRON-2100
On Thu, May 2, 2019 at 12:34 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > :-) > > I expect to have #2 out sometime today. > > On Thu, May 2, 2019, 12:11 PM Justin Leet <justinjl...@gmail.com> wrote: > >> > >> > I personally >> > don't like this feature gap in full dev. It seems Otto agrees, and >> Casey at >> > the very least sees it as enough of an issue to gate us from 0.8. >> > >> >> +1 on all of this. I don't like it either. >> >> >> > Our vote landed 2-2. We are having a discussion about what to do with >> the >> > release. This is that discussion. >> >> >> I'm going to be honest, my response was a combination of misreading what >> you said and thinking you were proposing delaying the release more >> seriously and feeling a bit blindsided by a perceived move from the >> initial >> "take more time than originally anticipated" (which in my head I took as a >> couple days) to "versus next week, or the week after" (where delaying >> things weeks is something I personally would like not buried so far down >> in >> the thread). Totally my bad, sorry about that. >> >> Other than that, it sounds like we're pretty much in agreement. >> >> Here's my current understanding of the state and consensus as of right now >> (which is subject to change as more discussion happens): >> >> - Most of the people in the thread are in favor of #2 for 0.7.1 and #3 >> for 0.8.0. >> - I don't believe I've seen an explicit response from Otto on what >> he >> thinks about doing this, and from a personal perspective like to >> see what >> his opinion is as the person who originally brought it up. >> - Mike said he's going to kick out a PR that addresses #2 >> - After that undergoes the normal review process and is merged, we >> proceed normally and cut RC2. >> >> >> On Thu, May 2, 2019 at 1:14 PM Michael Miklavcic < >> michael.miklav...@gmail.com> wrote: >> >> > I think your later point about using a project release version, from the >> > example of using other projects, is a valid one. To exactly that >> point, a >> > community member (Otto) brought up an issue/bug they found through >> testing >> > that they were previously unaware of and did not find documented. Which >> was >> > argued would be confusing to someone wanting to use a published >> release. We >> > discussed the implications of this bug/feature gap. And incidentally, it >> > sounds like some people see full dev as more useful than just a dev box, >> > others do not, independent of what we chose to name it. That came from >> our >> > discussion about it. >> > >> > The expectation I had from my discussion with the contributors was that >> > this fix for aggregation was ready. So to your point about whether it >> > belonged or not, I'm inclined to say yes, had it been ready. I >> personally >> > don't like this feature gap in full dev. It seems Otto agrees, and >> Casey at >> > the very least sees it as enough of an issue to gate us from 0.8. New >> > information about that feature has changed my mind about what to do >> about >> > it in the short term. I think we should move forward. >> > >> > Our vote landed 2-2. We are having a discussion about what to do with >> the >> > release. This is that discussion. >> > >> > On Thu, May 2, 2019, 10:52 AM Justin Leet <justinjl...@gmail.com> >> wrote: >> > >> > > @Mike >> > > I have a different question: Why is this enough to consider delaying a >> > > release in the first place for a fairly involved fix? >> > > >> > > There was a discuss thread, where the general agreement was that we >> had >> > > enough value to do a release (Over a month ago. And more things have >> gone >> > > into master since then). There's a good number of fixes, and not just >> > > trivial ones either. The general consensus here seems to be that the >> > > management UI issue is fairly minor for a point release (after all, >> > there's >> > > been multiple people who think option 2 is sufficient), but becomes >> > > important if we want to release a minor version. The question I asked >> > > myself about this was ""Does this issue detract enough value that a >> > release >> > > isn't worthwhile?" and my answer was, and still is, "No, we have >> enough >> > > value to do a meaningful release". >> > > >> > > I'm fine with delaying or cancelling a release because we find issues >> > that >> > > are severe enough or we don't think there's enough value anymore, but >> to >> > be >> > > entirely honest, I'm absolutely shocked this issue has blown up so >> much. >> > > However, if you want to have a discuss thread to reevaluate if it's >> > > worthwhile to do a release, go for it. The communities' calculus on >> the >> > > "Does this issue detract enough value that a release isn't >> worthwhile?" >> > may >> > > be different than mine. >> > > >> > > Having said all that, to a large extent, I think you're right. It >> really >> > > doesn't matter* that much* if we release next week or the week after >> or >> > > whenever. But at the same time I personally get super frustrated when >> I >> > go >> > > to use a project, find a bug, it's already known and fixed, but it >> just >> > > never puts out a released version. Every cutoff is largely arbitrary, >> > but >> > > I think getting our improvements and fixes out there is important. >> One of >> > > the things we've done fairly well is put out releases at a fairly >> decent >> > > cadence for a project this large. I really don't want to set the >> > precedent >> > > of just increasingly pushing out point releases for stuff like this. >> > > >> > > On Thu, May 2, 2019 at 12:52 PM Nick Allen <n...@nickallen.org> >> wrote: >> > > >> > > > I think any open source project needs to strive to cut releases >> > > regularly. >> > > > This is healthy for the project and community. It gets new features >> > and >> > > > functionality out to the community so we can get feedback, find >> what is >> > > > working and what is not, iterate and improve. You probably agree >> with >> > > > this. >> > > > >> > > > While releasing this week or next may not matter in the grand >> scheme, >> > if >> > > we >> > > > want to cut releases regularly, then we need to bear down and just >> do >> > it. >> > > > Case in point, I opened the initial discussion for this release on >> > March >> > > > 13th [1] and it is now May 2nd and we have yet to release 7 weeks >> > later. >> > > > >> > > > -- >> > > > [1] >> > > > >> > > > >> > > >> > >> https://lists.apache.org/thread.html/4f58649139f0aa6276f96febe1d0ecf9e6b3fb5b2b088cba1e3c4d81@%3Cdev.metron.apache.org%3E >> > > > >> > > > >> > > > On Thu, May 2, 2019 at 11:51 AM Michael Miklavcic < >> > > > michael.miklav...@gmail.com> wrote: >> > > > >> > > > > As a more general question, can I ask why we're feeling pressure >> to >> > > push >> > > > > out a release in the first place? Again, I'm happy to continue >> with >> > > > option >> > > > > 2. Let's move forward and get out the release. But is there a >> reason >> > > why >> > > > we >> > > > > think it has to get out now, versus next week, or the week after? >> > Otto >> > > > > pointed out a legitimate issue, dev environment or not, and I'm >> > unclear >> > > > why >> > > > > we have an issue with waiting for the fix. There's no pressure on >> > this, >> > > > > imho. >> > > > > >> > > > > On Thu, May 2, 2019, 9:12 AM Otto Fowler <ottobackwa...@gmail.com >> > >> > > > wrote: >> > > > > >> > > > > > I remember this now, but I’m not sure how I would have related >> this >> > > to >> > > > a >> > > > > > parser aggregation pr honestly. >> > > > > > >> > > > > > >> > > > > > On May 2, 2019 at 07:54:13, Shane Ardell ( >> shane.m.ard...@gmail.com >> > ) >> > > > > wrote: >> > > > > > >> > > > > > Here's a link to the ngrx discussion thread from a few months >> back: >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://lists.apache.org/thread.html/06a59ea42e8d9a9dea5f90aab4011e44434555f8b7f3cf21297c7c87@%3Cdev.metron.apache.org%3E >> > > > > > >> > > > > > On Thu, May 2, 2019 at 1:17 PM Otto Fowler < >> > ottobackwa...@gmail.com> >> > > > > > wrote: >> > > > > > >> > > > > > > If you can find a link in the archives for that thread, it >> would >> > > > really >> > > > > > > help. >> > > > > > > >> > > > > > > I don’t think sending them up as one sensor would work…. as >> > > something >> > > > > > > quick. I think it is an interesting idea from a higher level >> that >> > > > would >> > > > > > > need some more thought though ( IE: what if every sensor in >> the >> > ui >> > > > was >> > > > > a >> > > > > > > sensor group, and the existing entries where just groups of 1 >> ). >> > > > > > > >> > > > > > > As far as I can see, we have brought up the idea of a release >> > > > > ourselves, >> > > > > > I >> > > > > > > don’t see why we don’t just swarm this issue and get it right >> > then >> > > > > > release. >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > On May 2, 2019 at 04:16:31, Tamás Fodor ( >> ftamas.m...@gmail.com) >> > > > wrote: >> > > > > > > >> > > > > > > In PR#1360 we introduced a new state management strategy >> > involving >> > > a >> > > > > new >> > > > > > > module called Ngrx. We had a discussion thread on this a few >> > months >> > > > ago >> > > > > > and >> > > > > > > we successfully convinced you about the benefits. This is one >> of >> > > the >> > > > > > > reasons why this PR is going to be still huge after cleaning >> up >> > the >> > > > > > commit >> > > > > > > history. After you having a look at the changes and the >> feature >> > > > itself, >> > > > > > > there's likely have questions about why certain parts work as >> > they >> > > > do. >> > > > > > The >> > > > > > > thing what I'd like to point out is that, yes, it probably >> takes >> > > more >> > > > > > time >> > > > > > > to get it in. >> > > > > > > >> > > > > > > In order to being able to release the RC, wouldn't it be an >> easy >> > > and >> > > > > > quick >> > > > > > > fix on the backend if it sent the aggregated parsers to the >> > client >> > > as >> > > > > > they >> > > > > > > were one sensor? It's just an idea, it might be wrong, but at >> > least >> > > > we >> > > > > > > shouldn't have to wait until the aforementioned PR gets ready >> to >> > be >> > > > > > merged >> > > > > > > to the master. >> > > > > > > >> > > > > > > On Wed, May 1, 2019 at 4:16 PM Justin Leet < >> > justinjl...@gmail.com> >> > > > > > wrote: >> > > > > > > >> > > > > > > > Short version: I'm in favor of #2 of 0.7.1 and #1 as a >> blocker >> > > for >> > > > > > 0.8.0. >> > > > > > > > #3 seems like a total waste of time and effort. >> > > > > > > > >> > > > > > > > The wall of text version: >> > > > > > > > I agree this isn't "just the wrong thing shown", but for >> > > completely >> > > > > > > > different reasons. >> > > > > > > > >> > > > > > > > To be extremely clear about what the problem is: Our "dev" >> > > > > environment >> > > > > > > > (whose very name implies the audience is develops) uses a >> > > > > > > performance-based >> > > > > > > > advanced feature to ensure that all our of sample flows are >> > > > regularly >> > > > > > run >> > > > > > > > and produce data. This feature has a bare minimal >> > implementation >> > > to >> > > > > be >> > > > > > > > enabled via Ambari, which it currently is by default. This >> is >> > > > because >> > > > > > of >> > > > > > > > the limited resources available that previously resulted in >> us >> > > > > turning >> > > > > > > off >> > > > > > > > Yaf, and therefore testing it during regular full dev runs. >> > Right >> > > > now >> > > > > > > > however, this feature is not exposed through the management >> UI, >> > > and >> > > > > > > > therefore it isn't obvious what the implications are. Am I >> > > missing >> > > > > > > anything >> > > > > > > > here? >> > > > > > > > >> > > > > > > > For users actually choosing to use the parser aggregation >> > feature >> > > > in >> > > > > a >> > > > > > > > non-full-dev environment, I'd expect substantially more >> care to >> > > be >> > > > > > > involved >> > > > > > > > given the lack of easy configuration for it (after all, why >> > would >> > > > you >> > > > > > > > bother running the aggregated parser alongside the regular >> > > parser? >> > > > > This >> > > > > > > > could be more explicitly stated, but again that feels like a >> > doc >> > > > > > problem. >> > > > > > > > Right now I could essentially provide two of the same parser >> > and >> > > > > create >> > > > > > > the >> > > > > > > > same problem, so right now aggregation is only special >> because >> > it >> > > > > runs >> > > > > > on >> > > > > > > > dev by default). This is, in my opinion, primarily a first >> > > > impression >> > > > > > > > problem and likely one of many areas that could use improved >> > > > > > > documentation. >> > > > > > > > >> > > > > > > > Quite frankly, I think the issue pointed out here could >> mostly >> > be >> > > > > > > resolved >> > > > > > > > by documenting how the current aggregation is done in dev, >> and >> > > > > telling >> > > > > > > how >> > > > > > > > to change it. Especially for a 0.x.1 release, which is >> > primarily >> > > > bug >> > > > > > > > fixes. As can be inferred from my vote, I don't think this >> > > problem >> > > > > is a >> > > > > > > > problem that needs solving in a point release. I would >> support >> > > > > > improving >> > > > > > > > the documentation, both full-dev and for aggregation in >> general >> > > for >> > > > > the >> > > > > > > > 0.7.1 point release, while making a 0.8.0 release contingent >> > upon >> > > > the >> > > > > > > > outstanding PRs to enable it in the management UI. >> > > > > > > > >> > > > > > > > There are a couple deeper issues, imo, that I care >> > substantially >> > > > more >> > > > > > > about >> > > > > > > > than this in particular >> > > > > > > > * The dev environment is being used as our intro for users, >> > > because >> > > > > > it's >> > > > > > > > convenient for us to not maintain more environments (which >> has >> > > > been a >> > > > > > > major >> > > > > > > > pain point in the past). Worse, the dev environment strongly >> > > > implies >> > > > > > it's >> > > > > > > > for Metron developers, rather than people looking to build >> on >> > top >> > > > of >> > > > > > > > Metron. We need an actual strategy for providing end users a >> > > clean >> > > > > > > > impression of Metron (this could be clarifying what the >> > > > expectations >> > > > > of >> > > > > > > > full dev are, renaming it to something like "full-demo", >> > > something >> > > > > more >> > > > > > > > involved, etc.). This is something that we've needed for >> awhile >> > > in >> > > > > > > general, >> > > > > > > > and includes larger topics like improving our website, >> > > potentially >> > > > > > > > improving the site book, actually publishing our Javadocs >> > > somewhere >> > > > > so >> > > > > > > > people can develop things easier, publishing out info about >> > > Stellar >> > > > > > > > functions in a better manner, etc. >> > > > > > > > * The fact that parsers are handled in Ambari at all. It's >> > awful >> > > > and >> > > > > > > leads >> > > > > > > > to situations like this. To the best of my knowledge, once >> we >> > can >> > > > do >> > > > > > > > chaining and aggregation in the Management UI, we should be >> > able >> > > to >> > > > > > > > entirely divorce these two overlapping domains. I'd love to >> see >> > > > > parsers >> > > > > > > > ripped out of Ambari, then full-dev manages all the setup >> via >> > > REST. >> > > > > At >> > > > > > > that >> > > > > > > > point, we can easily tell everyone to just use the >> management >> > UI. >> > > > > > > > >> > > > > > > > On Wed, May 1, 2019 at 7:23 AM Otto Fowler < >> > > > ottobackwa...@gmail.com> >> > > > > > > > wrote: >> > > > > > > > >> > > > > > > > > I think it would help if the full consequences of having >> the >> > UI >> > > > > show >> > > > > > > the >> > > > > > > > > wrong status where listed. >> > > > > > > > > >> > > > > > > > > Someone trying metron, will, by default , see the wrong >> thing >> > > in >> > > > > the >> > > > > > UI >> > > > > > > > for >> > > > > > > > > the ONLY sensors they have that are running and doing >> data. >> > > > > > > > > >> > > > > > > > > What happens when they try to start them to make them >> work? >> > > One, >> > > > > two >> > > > > > or >> > > > > > > > > all? >> > > > > > > > > What happens when he edits them or try to add >> > transformations? >> > > > One, >> > > > > > two >> > > > > > > > or >> > > > > > > > > all? >> > > > > > > > > What other things can you do with the sensors in the ui? >> What >> > > > > > happens? >> > > > > > > > > >> > > > > > > > > Are we recommending aggregation on the list and elsewhere >> for >> > > > > users? >> > > > > > > Are >> > > > > > > > > we recommending something that is going to ensure they get >> > into >> > > > > this >> > > > > > > > > situation? >> > > > > > > > > >> > > > > > > > > I think this is more than ‘just the wrong thing shown’ in >> the >> > > ui. >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > On April 30, 2019 at 20:48:10, Michael Miklavcic ( >> > > > > > > > > michael.miklav...@gmail.com) wrote: >> > > > > > > > > >> > > > > > > > > The vote for RC1 did not pass and I'd like to kickstart >> some >> > > > > > discussion >> > > > > > > > > about what we should do. >> > > > > > > > > >> > > > > > > > > I started taking a look at PR#1360 and it looks like this >> > isn't >> > > > > quite >> > > > > > > as >> > > > > > > > > close to being able go in as I had originally expected. I >> > want >> > > to >> > > > > > talk >> > > > > > > > > about options here. It seems to me that we can: >> > > > > > > > > >> > > > > > > > > 1. Wait for PR#1360 to go in, but this is likely going to >> > take >> > > > more >> > > > > > > time >> > > > > > > > > than originally anticipated >> > > > > > > > > 2. Accept the issue in full dev, but add some notes in the >> > > > > developer >> > > > > > > > > docs about the current feature gap and why sensors aren't >> > > showing >> > > > > > > status >> > > > > > > > in >> > > > > > > > > the management UI when aggregation is enabled. >> > > > > > > > > 3. Find some other workable UI solution. >> > > > > > > > > 4. Other option? >> > > > > > > > > >> > > > > > > > > All things considered, I'm personally leaning towards #2 >> in >> > the >> > > > > > > > short-term, >> > > > > > > > > but I think we should probably talk about this a bit >> before >> > > > > deciding >> > > > > > > what >> > > > > > > > > RC2 should be. >> > > > > > > > > >> > > > > > > > > Best, >> > > > > > > > > Mike >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> >
