Looks good, thanks!
On August 15, 2018 at 19:38:12, Ryan Merriman (merrim...@gmail.com) wrote: Otto, I believe the items you requested are in the feature branch now. Is there anything outstanding that we missed? The Jiras for the Pcap feature branch should be up to date: https://issues.apache.org/jira/browse/METRON-1554 On Mon, Aug 13, 2018 at 5:13 PM, Ryan Merriman <merrim...@gmail.com> wrote: > - Date range limits on queries > > I will add a warning in the Job cleanup PR. That seems like an > appropriate place for it (ie. make sure you don't cause health issues in > your cluster). > > - UI should manage a queue/history of jobs > > I can add some documentation around killing jobs manually with the YARN > CLI. However if they haven't set up a YARN queue, I'm not sure how you > would view only Pcap jobs. I'm also not sure how you would get the > application id for the job to kill because it's not displayed anywhere in > the UI. However, I believe we are wired for a job name but REST doesn't > set this. Maybe we could get a proper job name associated with pcap > queries and then this would be possible to document? > > - Documentation/blueprint for YARN configuration > > You make a good point. A YARN tuning guide for Metron does sound useful. > I will add a follow on Jira. > > On Mon, Aug 13, 2018 at 4:53 PM, Otto Fowler <ottobackwa...@gmail.com> > wrote: > >> >> - Date range limits on queries >> >> I took the point the wrong way apparently, sorry, I withdraw. I thought >> you meant allow specifying a limit on the query, not the system imposing a >> limit. >> This should be documented with a warning or something >> >> - UI should manage a queue/history of jobs >> >> I was thinking that if there where multiple users/jobs, there should >> be some thought or documentation + script on how to manage them. >> “To see all the jobs still running on your cluster, across users and ui >> instances do XXXXX” >> “If there is an issue with the jobs you can’t resolve in the UI for that >> user, or you are an admin and want to do something then XXXXX" >> >> - Documentation/blueprint for YARN configuration >> >> I agree with what you are saying. Although, we offer guidance on storm >> tuning, and that is conceptually the same isn’t it? That is why it comes >> to mind. >> Maybe this can be a follow on, in the tuning guide? >> >> On August 13, 2018 at 17:36:41, Ryan Merriman (merrim...@gmail.com) >> wrote: >> >> - Date range limits on queries >> >> Can you describe what you think is needed here? Each Metron user could >> have different volumes of pcap data spread out over different time >> periods. Are you saying we should limit the data range to something either >> >> constant or configurable? Are we sure all users would want this? Am I >> misinterpreting this requirement? >> >> - UI should manage a queue/history of jobs >> >> What should we document here? Reading that bullet point again, it's sort >> of vague and not very description. What I am referring to is a design that >> >> provides users a way to view and manage jobs in the UI. Currently jobs can >> >> only be run 1 at a time and progress is shown with a status bar, so it's >> somewhat interactive. >> >> - Documentation/blueprint for YARN configuration >> >> >
