Hi All, Perhaps you also want to have a look at OPNSense. I think others also mentioned it before.
It is mature and stable. It is based on hardened free-bsd. I have worked with Ansible Modules to automate its deployment and I can see a lot of benefits that would make integration easy. For example its configuration is robust and comes from a single file that can recreate the whole firewall configuration: Routes, VPNs, Certificate Authorities, NAT Rules, DHCP Servers, Reservations, Resolvers, Optional Packages, Intrusion Detection, etc. Some functions, like Rules, are available via API. I know the project team and they are working on improving API so that all the firewall options are available also via API. Perhaps there is a natural fit for ACS. Rafael On Mon, 2021-08-16 04:54 PM, Pierre-Luc Dion <pdion...@apache.org> wrote: > Hi Rohit, > > So look like OpenWRT could be a good alternative then, but how would you > handle other services such as load-balancing, vpn,...? would that be from > another Service VMs, kind of doing service mesh ? > > Cheers, > > > > On Mon, Aug 16, 2021 at 5:37 AM Rohit Yadav " > target="_blank"><rohit.ya...@shapeblue.com> > wrote: > > > Hi PL, > > > > I tested OpenWRT on KVM, and I don't see any performance loss in terms of > > bandwidth/IO and memory, in fact it seems to be faster than the Debian > > based systemvmtemplate due to: > > > > * Smaller kernel size (the Debian kernel includes all sorts of device > > drivers for sound and what not which is not necessary for VR/systemvm) > > * Faster ssh (the dropbear ssh kernel is on-par with openssh-server > > but has smaller footprint and missing features such as sftp that we don't > > need in VR/systemvm; the ssh and programming seemed really fast compared to > > Debian's openssh-server) > > * Really low memory consumption (Debian10+ eats up some RAM due to > > non-optimised services, esp systemd compared to OpenWRT) > > * Lua based really-fast APIs and bus (openwrt has > > https://openwrt.org/docs/techref/ubus, I'm looking into how we can build > > something similar for our Debian based systemvmtemplate if migrating to > > OpenWRT is going to be a pain or unacceptable by the community) > > > > Based on your comment I found that VyOS rolling release has API support ( > > https://blog.vyos.io/vyos-rolling-release-has-got-an-http-api) however > > that is probably not supported for their lts/legacy release, we'll also > > face issue with installing/configuring custom packages (java etc) that we > > need for systemvm/VR. > > > > > > Regards. > > > > ________________________________ > > From: Pierre-Luc Dion " target="_blank"><pdion...@apache.org> > > Sent: Friday, August 13, 2021 21:51 > > To: dev " target="_blank"><dev@cloudstack.apache.org> > > Subject: Re: [DISCUSS] NextGen VR? > > > > From a certain point of view, OpenWRT is most likely not suitable for our > > needs, this project is highly oriented for router appliance platforms and > > kernel is most likely highly optimised for a small memory footprint and > > might not have performance and virtualization in mind. > > > > It's a shame that there is not some sort of API driven open source VyOS :-S > > > > Anything useful that would come out of the CloudNative projects? > > > > > > On Fri, Aug 13, 2021 at 5:48 AM Rohit Yadav " > > target="_blank"><rohit.ya...@shapeblue.com> > > wrote: > > > > > Hi Pierre-Luc, > > > > > > Thanks for replying. > > > > > > We've another thread on-going to discuss IPv6 support where we've indeed > > > discussed idea of introducing dynamic routing capabilities in the VR in a > > > future phase with FRR in the VR. > > > > > > For the VR agent idea, I'm indeed thinking of API driven way (the agent > > > exposes an API for CloudStack and a custom CLI perhaps) for it to manage > > > iptables, software packages/services such as dnsmasq etc. Right now it's > > > possible to configure some of the services in network offering (for ex. > > if > > > you don't want LB etc). > > > > > > I've indeed looked at some alternatives: > > > > > > * Alpine: I experimented a new systemvmtemplate based on Alpine ( > > > https://github.com/shapeblue/alpine-systemvm) but we saw many cons with > > > Alpine and the gains weren't significant so we decided to drop the idea > > of > > > moving from Debian to Alpine. > > > * VyOS: I've concerns about project and community (I could be wrong) > > > to have our VR depend on it as the "default" provider; though I see an > > old > > > PR > > > * OpenWRT: Among available router distributions, I really liked it - > > > it has (a) a good health and active community and commit activity ( > > > https://github.com/openwrt/openwrt/graphs/contributors), (b) multi-years > > > (10+yrs) of releases (https://openwrt.org/about/history), (c) a > > > configuration system (UCI) and UI (LUCI) which can be used to manage > > > services, interfaces, firewall, routes etc with a Lua based API, (d) opkg > > > packaging system, most software packages or equivalent are available that > > > our Debian based systemvmtemplate needs/uses (java jre workaround from > > > Alpine though), (e) good documentation on extending features/writing > > > plugins etc, (f) they tend to do regular releases and use Linux/LTS > > kernel. > > > * pfSense/opnSense: the major cons are FreeBSD-based kernel and > > skills > > > that CloudStack community may not have in terms of debugging, > > investigation > > > etc like they have for Linux > > > > > > Among the above, OpenWRT is probably worth exploring in my opinion; other > > > some firewall/iptables and iproute2 (interfaces and routes) API/library > > > that can be used by the VR agent on Debian to program the firewall (acls, > > > firewall, pf), IP/interface management and routing features in the VR > > > (currently it's all done in a manual sense which is untested and often > > > source of bugs and issues). > > > > > > Lastly, we already have this PR which aims to do automatic > > > systemvmtemplate registeration/installation/seeding and handle upgrades > > > (targetted for 4.16): https://github.com/apache/cloudstack/pull/4329 The > > > PR also unifies the systemvm template use to be extended for CKS use-case > > > (i.e. no separate template installation required for CKS). > > > > > > > > > Regards. > > > > > > ________________________________ > > > From: Pierre-Luc Dion " target="_blank"><pdion...@apache.org> > > > Sent: Thursday, August 12, 2021 16:45 > > > To: dev " target="_blank"><dev@cloudstack.apache.org> > > > Subject: Re: [DISCUSS] NextGen VR? > > > > > > Hi Rohit, > > > > > > Like it! Our VR system is due for some rethinking! > > > I don't have much points to add to issues you highlight, it seems pretty > > > complete. > > > > > > Here are some more features or ideas that would be interesting to see in > > a > > > new VR system: > > > - Use or support for routing protocol, such as BGP or OSBP so we could > > > provide a more dynamic PrivateGateway concept. using FRR[1]? > > > - have an API driven way to configure IPtables and other network > > services. > > > - could we decouple network services such as LB, VPNserver, gateways > > from > > > the VR ? > > > > > > Debian has been a pain for building VR because the iso defined in our > > > config need constant update, but on the other hand it's been proved to > > be a > > > reliable OS, we saw some of our VR with uptime over 1000 days. > > > > > > As an alternative OS, I'd be curious to look at VyOS[2] or Alpine Linux. > > > > > > From a certain perspective, us providing the systemvm template make sure > > > that systemVM will deploy. work reliably and make it a single template to > > > test. Compared to a system that would just provide RPM/DEB packages and > > > mechanism to push configs, this could require to test all kinds of > > template > > > scenarios, since users could use any version of distro to deploy their > > > systemVM/VRs. > > > > > > "Users forget to register the right systemvmtemplate for a new ACS > > version > > > and upgrades fail": Maybe we could automatically register the new > > template > > > during the upgrade process? This feature used to exist in the Citrix > > > CloudPlatform. > > > > > > > > > [1] https://frrouting.org/ > > > [2] https://github.com/vyos/vyos-1x > > > > > > On Wed, Aug 11, 2021 at 12:37 PM Rohit Yadav " > > > target="_blank"><rohit.ya...@shapeblue.com> > > > wrote: > > > > > > > All, > > > > > > > > We've over the years create a VR that largely is stable but we've > > > > discussed the pain of maintaining, extending, and upgrading VRs both in > > > > lists and in user-groups and CCC conferences. > > > > > > > > On a high-level the pain points are: > > > > > > > > * It's difficult to debug, investigate VR for operators and support > > > > team > > > > * Maintaining the VR code, fixing bugs, implementing features is a > > > > pain for developers; further the xml&json databags based programming > > > model > > > > is confusing > > > > * Any fix or changes requires a new systemvmtemplate or VR codebase > > > > whose patching requires restarting the VR or destroying an old VR > > > > * No uniform VR programming API (current approach is SSH+databags > > and > > > > execute a script), this makes testing VR difficult and QA in isolation > > is > > > > not possible > > > > * Users forget to register the right systemvmtemplate for a new ACS > > > > version and upgrades fail > > > > * Others (please share yours)? > > > > > > > > Among these pain points my colleagues have proposed a PR targeting 4.16 > > > > [1] that aims to unify systemvmtemplate as a building block that is > > > bundled > > > > as part of ACS rpm/deb/* packages which CloudStack will automatically > > > > seed/register/use with which upgrades will be as simple as a yum update > > > or > > > > apt-get upgrade. Further, my colleagues and I are exploring a live > > patch > > > > API which in near future that can patch a running systemvm/VR using > > > > systemvm.iso (or deprecate systemvm.iso and use ssh/scp to patch?) > > > without > > > > requiring to reboot/recreate it. Hopefully, this addresses some of > > those > > > > pain points. We request the community for your feedback and > > > > review/participation in the PR. > > > > > > > > Open questions, topics to discuss and gather feedback: > > > > > > > > * VR programming: Should we explore a new light-weight VR agent > > that > > > > provides an API (restful/grpc, or CLI?), some mechanism of live > > patching > > > VR > > > > code, packages, and kernel? > > > > * Refactoring isolated network and VPC codebases into a unified > > > > codebase and feature sets (assumption that isolated network are > > largely a > > > > VPC with single tier), does it benefit the community, users, and > > > developers? > > > > * Underlying OS: > > > > * should we consider something other than Debian, any > > suggestions? > > > > * or explore stable/widely used and maintained opensource router > > > > distributions such as OpenWRT [2] which ships with a UI and > > > > CLI/configuration system UCI [3]? The cons of the approach are a new > > > > dependency and some likely missing packages. > > > > * In current VR codebase, most of the effort is spent in > > > > implementing/maintaining router packages/configure codebase which we > > can > > > > get rid of by depending on a stable Linux router distro which ships > > with > > > > some API/config system. Not choosing an existing router distribution > > > means > > > > we continue to DIY router programming+config management codebase. > > > > * Any other ideas? > > > > > > > > Thoughts, feedback? > > > > > > > > [1] https://github.com/apache/cloudstack/pull/4329 > > > > [2] https://github.com/openwrt/openwrt/graphs/contributors > > > > [3] https://openwrt.org/docs/guide-user/base-system/uci > > > > > > > > > > > > Regards. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
