Re: [Zeek-Dev] Zeek Supervisor Command-Line Client
>> Suggestion: `zeekcl`, Zeek (Command-Line) CLlient. > > "zeekcl" is very close to "zeekctl", which could lead to confusion. > "zcl" maybe? > >> Is use of Python still desirable for other reasons? Otherwise, I >> lean >> towards `zeekcl` being C++. > > No particular preference from my side, I can see either. Effort is > probably about the same in this model, and C++ does have the advantage > of less dependency issues. I agree - I actually kind of like the idea that zeekcl does not have python as a dependency. >> I plan to have `zeekcl` code/tests live inside the main Zeek repo. > > Makes sense to me as well. Agreed here too. Johanna ___ Zeek-Dev mailing list Zeek-Dev@zeek.org http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
Re: [Zeek-Dev] Zeek Supervisor Command-Line Client
On Thu, Jun 18, 2020 at 7:45 AM Vlad Grigorescu wrote: > My main concern was Broker version incompatibilities between the > newly-installed zcl, and the running cluster, which I think would be > addressed by that (i.e. to stop a cluster, you stop the supervisor service on > the manager, and then the other services will lose their connection and also > stop). A clarification that may help you: the "orphaning" behavior isn't related to Broker connections, it's related to the parent-child relationship between processes. So there's a process tree here with `zeek` in supervisor-mode at the root and child processes that are individual cluster nodes (worker, manager, logger, proxy). The normal termination behavior for the supervisor process is to gracefully kill and wait for all children to exit. In the very exceptional case of the supervisor exiting/crashing without having cleaned up all children, those children will self-terminate upon noticing they are no longer parented to the supervisor. - Jon ___ Zeek-Dev mailing list Zeek-Dev@zeek.org http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
Re: [Zeek-Dev] Zeek Supervisor Command-Line Client
On Thu, Jun 18, 2020 at 12:11 AM Robin Sommer wrote: > For (1), the above applies: we'll rely on standard sysadmin processes > for updating. That means you'd use "zeekcl" to shutdown the cluster > processes, then run "yum update" (or whatever), then use "zeekcl" > again to start things up again. (The Zeek supervisor will be running > already at that point, managaged through systemd or whatever you're > using). I have a slightly different take: isn't it more common to expect "start" and "stop" operations here to be done by the service-manager rather than Zeek client? I'm assuming "update/deploy Zeek installation" could involve a change in the `zeek` binary and that implements the supervisor process itself, so you'd want, at the level of system services, to stop the entire Zeek process tree, including the root supervisor. That doesn't exclude the possibility of the client having operations like "start" (spawn `zeek -j `), "stop" (kill the root `zeek` supervisor process), or even others that dynamically add/remove cluster nodes from the tree, but that's probably not the common/expected usage to prioritize since it's again back to model of the process tree being managed manually by the user, independent from a system's service-manager. - Jon ___ Zeek-Dev mailing list Zeek-Dev@zeek.org http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
Re: [Zeek-Dev] Zeek Supervisor Command-Line Client
Thanks Robin, that helps. On Thu, Jun 18, 2020 at 2:11 AM Robin Sommer wrote: > > There are two parts here: (1) deploying the Zeek installation itself, > and (2) deploying any configuration changes (incl. new Zeek scripts). > > For (1), the above applies: we'll rely on standard sysadmin processes > for updating. That means you'd use "zeekcl" to shutdown the cluster > processes, then run "yum update" (or whatever), then use "zeekcl" > again to start things up again. (The Zeek supervisor will be running > already at that point, managaged through systemd or whatever you're > using). > > (2) is still a bit up in the air. With 3.2, there won't be any support > for distributing configurations automatically, but we could add that > so that config files/scripts/packages do get copied around over > Broker. Feedback would be appreciated here: What's better, having > zeekcl manage that, or leave it to standard sysadmin process as well? > I re-read the design doc, and I think that the part I missed the first time through was suicide on orphaning. (Side-note: Given the much-needed trend towards bias-free terminology in technology, perhaps there's a better term here). My main concern was Broker version incompatibilities between the newly-installed zcl, and the running cluster, which I think would be addressed by that (i.e. to stop a cluster, you stop the supervisor service on the manager, and then the other services will lose their connection and also stop). I'm still a bit unclear on how to start a cluster. In my mind, where simply using the standard process/job control falls short is the need to operate across multiple physical systems. So, would that be a job for zcl? Or would the desired goal be that I have my, say, systemd unit set to constantly be restarting Zeek on my worker systems? If it can't connect to the manager, it would presumably immediately die given the orphaned state. The more tightly we couple the nodes together, the more quickly it'll detect failures, but the more sensitive it will be to flapping and unnecessary restarts. The cluster is relatively fragile right now (e.g. a manager node going away even for a brief period of time tends to lead to a crash, as on an even relatively busy system, as the backlog won't clear as timers and other events stack up). So I think that if we're moving cluster supervision out of a parallel process in `zeekctl cron` and into Zeek itself, we'll need to improve error detection and graceful recovery where possible. --Vlad ___ Zeek-Dev mailing list Zeek-Dev@zeek.org http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
Re: [Zeek-Dev] Zeek Supervisor Command-Line Client
On Thu, Jun 18, 2020 at 03:32 +, Vlad Grigorescu wrote: > As a concrete example, what does a cluster upgrade look like? The idea is to handle this more like other system services: you'll be in charge of getting the new Zeek version onto all your systems yourself, using whatever method you use for other software as well. For example, if you're installing through a package manager, you'd just run "update" on all systems. If you're installing from source, you'll either need to compile on each system, or copy the installation over manually. The underlying assumption is that people will already have a mechanism in place for administration of their systems, and we shouldn't be trying to reinvent the wheel, as ZeekControl oddly does. From a sysadmin perspective, ZeekControl is really doing a lot more right now that it should be doing; other tools don't work that way. We don't want it look like an APT anymore (https://github.com/zeek/zeek/issues/259). :-) > Today, that means install the new version on the manager, and then do > `zeekctl deploy`, which copies the files to the nodes and restarts the > cluster. All of that is done without Broker. There are two parts here: (1) deploying the Zeek installation itself, and (2) deploying any configuration changes (incl. new Zeek scripts). For (1), the above applies: we'll rely on standard sysadmin processes for updating. That means you'd use "zeekcl" to shutdown the cluster processes, then run "yum update" (or whatever), then use "zeekcl" again to start things up again. (The Zeek supervisor will be running already at that point, managaged through systemd or whatever you're using). (2) is still a bit up in the air. With 3.2, there won't be any support for distributing configurations automatically, but we could add that so that config files/scripts/packages do get copied around over Broker. Feedback would be appreciated here: What's better, having zeekcl manage that, or leave it to standard sysadmin process as well? > Reading the script linked in [2], I notice that zeekcl would not support > copying files from one node to another? Correct right now, (2) may or may not change that. > zeekctl print "print" will be supported (roadmap says not in 3.2 yet, but it should be easy to do, maybe we can get it in still). > zeekctl exec. "exec" will likely not be supported. We *could* support it, no technical reason for not doing that over Broker. It just s seems like another things that's better handled with different tools. Robin -- Robin Sommer * Corelight, Inc. * ro...@corelight.com * www.corelight.com ___ Zeek-Dev mailing list Zeek-Dev@zeek.org http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
Re: [Zeek-Dev] Zeek Supervisor Command-Line Client
> Suggestion: `zeekcl`, Zeek (Command-Line) CLlient. "zeekcl" is very close to "zeekctl", which could lead to confusion. "zcl" maybe? > Is use of Python still desirable for other reasons? Otherwise, I lean > towards `zeekcl` being C++. No particular preference from my side, I can see either. Effort is probably about the same in this model, and C++ does have the advantage of less dependency issues. > Zeek's scripting language (e.g. `ctl.zeek`), but I don't suggest that Ack, agree. > I plan to have `zeekcl` code/tests live inside the main Zeek repo. Makes sense to me as well. Robin -- Robin Sommer * Corelight, Inc. * ro...@corelight.com * www.corelight.com ___ Zeek-Dev mailing list Zeek-Dev@zeek.org http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
