We discovered a bug in our testing and felt it should be fixed before we merge. There is a PR up for review that already has a +1: https://github.com/apache/metron/pull/1168. I don't anticipate this changing anyone's vote but wanted to be clear about the state of the branch. If anyone is concerned with this and would like more discussion before we merge, let me know.
On Thu, Aug 16, 2018 at 8:25 AM, James Sirota <jsir...@apache.org> wrote: > +1 on the merge as well > > 16.08.2018, 05:46, "Casey Stella" <ceste...@gmail.com>: > > I'm +1 on the merge. This is great work and congrats to those who > > contributed to it! > > > > On Thu, Aug 16, 2018 at 8:27 AM Otto Fowler <ottobackwa...@gmail.com> > wrote: > > > >> Looks good, thanks! > >> > >> On August 15, 2018 at 19:38:12, Ryan Merriman (merrim...@gmail.com) > wrote: > >> > >> Otto, I believe the items you requested are in the feature branch now. > Is > >> there anything outstanding that we missed? The Jiras for the Pcap > feature > >> branch should be up to date: > >> https://issues.apache.org/jira/browse/METRON-1554 > >> > >> On Mon, Aug 13, 2018 at 5:13 PM, Ryan Merriman <merrim...@gmail.com> > >> wrote: > >> > >> > - Date range limits on queries > >> > > >> > I will add a warning in the Job cleanup PR. That seems like an > >> > appropriate place for it (ie. make sure you don't cause health > issues in > >> > your cluster). > >> > > >> > - UI should manage a queue/history of jobs > >> > > >> > I can add some documentation around killing jobs manually with the > YARN > >> > CLI. However if they haven't set up a YARN queue, I'm not sure how > you > >> > would view only Pcap jobs. I'm also not sure how you would get the > >> > application id for the job to kill because it's not displayed > anywhere in > >> > the UI. However, I believe we are wired for a job name but REST > doesn't > >> > set this. Maybe we could get a proper job name associated with pcap > >> > queries and then this would be possible to document? > >> > > >> > - Documentation/blueprint for YARN configuration > >> > > >> > You make a good point. A YARN tuning guide for Metron does sound > useful. > >> > I will add a follow on Jira. > >> > > >> > On Mon, Aug 13, 2018 at 4:53 PM, Otto Fowler < > ottobackwa...@gmail.com> > >> > wrote: > >> > > >> >> > >> >> - Date range limits on queries > >> >> > >> >> I took the point the wrong way apparently, sorry, I withdraw. I > thought > >> >> you meant allow specifying a limit on the query, not the system > imposing > >> a > >> >> limit. > >> >> This should be documented with a warning or something > >> >> > >> >> - UI should manage a queue/history of jobs > >> >> > >> >> I was thinking that if there where multiple users/jobs, there should > >> >> be some thought or documentation + script on how to manage them. > >> >> “To see all the jobs still running on your cluster, across users > and ui > >> >> instances do XXXXX” > >> >> “If there is an issue with the jobs you can’t resolve in the UI for > that > >> >> user, or you are an admin and want to do something then XXXXX" > >> >> > >> >> - Documentation/blueprint for YARN configuration > >> >> > >> >> I agree with what you are saying. Although, we offer guidance on > storm > >> >> tuning, and that is conceptually the same isn’t it? That is why it > comes > >> >> to mind. > >> >> Maybe this can be a follow on, in the tuning guide? > >> >> > >> >> On August 13, 2018 at 17:36:41, Ryan Merriman (merrim...@gmail.com) > >> >> wrote: > >> >> > >> >> - Date range limits on queries > >> >> > >> >> Can you describe what you think is needed here? Each Metron user > could > >> >> have different volumes of pcap data spread out over different time > >> >> periods. Are you saying we should limit the data range to something > >> either > >> >> > >> >> constant or configurable? Are we sure all users would want this? Am > I > >> >> misinterpreting this requirement? > >> >> > >> >> - UI should manage a queue/history of jobs > >> >> > >> >> What should we document here? Reading that bullet point again, it's > sort > >> >> of vague and not very description. What I am referring to is a > design > >> that > >> >> > >> >> provides users a way to view and manage jobs in the UI. Currently > jobs > >> can > >> >> > >> >> only be run 1 at a time and progress is shown with a status bar, so > it's > >> >> somewhat interactive. > >> >> > >> >> - Documentation/blueprint for YARN configuration > >> >> > >> >> > >> > > > ------------------- > Thank you, > > James Sirota > PMC- Apache Metron > jsirota AT apache DOT org > >
