- Date range limits on queries I will add a warning in the Job cleanup PR. That seems like an appropriate place for it (ie. make sure you don't cause health issues in your cluster).
- UI should manage a queue/history of jobs I can add some documentation around killing jobs manually with the YARN CLI. However if they haven't set up a YARN queue, I'm not sure how you would view only Pcap jobs. I'm also not sure how you would get the application id for the job to kill because it's not displayed anywhere in the UI. However, I believe we are wired for a job name but REST doesn't set this. Maybe we could get a proper job name associated with pcap queries and then this would be possible to document? - Documentation/blueprint for YARN configuration You make a good point. A YARN tuning guide for Metron does sound useful. I will add a follow on Jira. On Mon, Aug 13, 2018 at 4:53 PM, Otto Fowler <ottobackwa...@gmail.com> wrote: > > - Date range limits on queries > > I took the point the wrong way apparently, sorry, I withdraw. I thought > you meant allow specifying a limit on the query, not the system imposing a > limit. > This should be documented with a warning or something > > - UI should manage a queue/history of jobs > > I was thinking that if there where multiple users/jobs, there should > be some thought or documentation + script on how to manage them. > “To see all the jobs still running on your cluster, across users and ui > instances do XXXXX” > “If there is an issue with the jobs you can’t resolve in the UI for that > user, or you are an admin and want to do something then XXXXX" > > - Documentation/blueprint for YARN configuration > > I agree with what you are saying. Although, we offer guidance on storm > tuning, and that is conceptually the same isn’t it? That is why it comes > to mind. > Maybe this can be a follow on, in the tuning guide? > > On August 13, 2018 at 17:36:41, Ryan Merriman (merrim...@gmail.com) wrote: > > - Date range limits on queries > > Can you describe what you think is needed here? Each Metron user could > have different volumes of pcap data spread out over different time > periods. Are you saying we should limit the data range to something either > > constant or configurable? Are we sure all users would want this? Am I > misinterpreting this requirement? > > - UI should manage a queue/history of jobs > > What should we document here? Reading that bullet point again, it's sort > of vague and not very description. What I am referring to is a design that > > provides users a way to view and manage jobs in the UI. Currently jobs can > > only be run 1 at a time and progress is shown with a status bar, so it's > somewhat interactive. > > - Documentation/blueprint for YARN configuration > >
