Hi Federico Capoano, Zerotier network carrying mainly Management Traffic and RADIUS traffic. I have Openwisp as a controller, and My Portal + FreeRADIUS on another Server. Now my customer does not interact with Openwisp. only API do. So I can customize as much as possible. but i thought this might be an actual issue. why not drop everything on the mailing list, sorry for copy paste I am very lazy.. actually it took some time to reach this final version.
What type should I choose ? Bug or something else? Regards, Muhammad Ismail. On Fri, Apr 17, 2026 at 7:45 PM 'Federico Capoano' via OpenWISP < [email protected]> wrote: > Hi Mohammed Ismail, > > Can you clarify if you're using Zerotier for management or for > transporting traffic? > > I think it's worth creating an issue, but please do not paste this text as > is because it's not suited for the issue tracker. Follow the feature > request template at this URL: > https://github.com/openwisp/openwisp-controller/issues > > Best regards > *Federico Capoano* > OpenWISP OÜ > Kotkapoja tn 2a-10, 10615, Harju maakond, Tallinn, Estonia > VAT: EE101989729 > +372 59361689 > *openwisp.io* <http://openwisp.io> > > > On Thu, 16 Apr 2026 at 18:55, Mohammed Ismail <[email protected]> wrote: > >> ## Subject: ZeroTier route generation from IPAM child subnets — PoC and >> discussion >> >> Hi everyone, >> >> We run a multi-tenant WiFi management platform on top of OpenWISP with a >> growing ZeroTier deployment. We ran into multicast scaling issues with a >> flat `/8` route and built a working PoC — sharing it here to see if >> there's interest in upstreaming something like this. >> >> ### The Problem >> >> The ZeroTier backend currently generates a single route from the VPN's >> IPAM subnet: >> >> ```python >> config["routes"] = [{"target": str(self.subnet), "via": ""}] >> ``` >> >> ZeroTier uses ARP emulation — it converts ARP broadcasts into targeted >> multicast groups, one per IPv4 address. This works well, but the **Multicast >> Recipient Limit** (default 32) means that in a large flat network, >> multicast delivery is capped. With a `/8` route and all members in one >> broadcast domain, the multicast group for any given IP can't reach all >> potential peers when the network exceeds the recipient limit. This causes >> intermittent connectivity issues as ARP responses don't reliably reach all >> members. >> >> Per-server `/24` routes reduce the L2 scope per subnet. With ~250 >> members per `/24` instead of the entire network in one flat `/8`, the >> multicast groups stay within manageable bounds and ARP resolution works >> reliably. >> >> Additionally: >> - `vpn_backends.py` removes `routes` from the ZeroTier JSON schema (no >> UI exposure) >> - `_get_repsonse()` strips `routes` and `ipAssignmentPools` from ZT API >> responses (no observability) >> >> ### What We Did — PoC on zerotier_service.py >> >> We modified `_add_routes_and_ip_assignment()` to generate routes from >> IPAM child subnets instead of just the parent. Here's the diff: >> >> ```diff >> - Adds ZeroTier network routes >> - and IP assignmentpools through OpenWISP subnet >> + Adds ZeroTier network routes from IPAM child subnets (if any) >> + and IP assignment pools through OpenWISP subnet. >> + >> + If the VPN's IPAM subnet has child subnets, generates routes for >> + both the parent (cross-subnet routing) and each child (ARP >> isolation). >> + Otherwise falls back to single route from parent subnet. >> >> - config["routes"] = [{"target": str(self.subnet), "via": ""}] >> + try: >> + from openwisp_ipam.models import Subnet >> + parent = >> Subnet.objects.filter(subnet=str(self.subnet)).first() >> + if parent: >> + children = list(parent.child_subnet_set.only('subnet')) >> + if children: >> + # Parent route ensures cross-subnet reachability >> + # Child routes (more specific) provide ARP isolation >> + config["routes"] = [ >> + {"target": str(self.subnet), "via": ""} >> + ] + [ >> + {"target": str(child.subnet), "via": ""} >> + for child in children >> + ] >> + else: >> + config["routes"] = [{"target": str(self.subnet), >> "via": ""}] >> + else: >> + config["routes"] = [{"target": str(self.subnet), "via": >> ""}] >> + except Exception: >> + # Fallback: original behavior if anything fails >> + config["routes"] = [{"target": str(self.subnet), "via": ""}] >> ``` >> >> This gives us a parent `/8` route (cross-subnet reachability) plus >> per-server `/24` child routes. >> >> **Important caveat about L2 isolation**: The parent `/8` route with `via: >> ""` (directly attached) means devices still consider the entire `/8` as >> L2-local. For cross-subnet traffic, devices would still ARP through the >> flat `/8`, hitting the multicast recipient limit. In our deployment this >> doesn't matter — our traffic is entirely intra-/24 (hotspots only talk to >> their server's RADIUS at `.1` and controller at `.254`). But for >> deployments that need cross-subnet communication, the parent route should >> use a gateway (`"via": "<router_zt_ip>"`) instead of being directly >> attached. We kept `via: ""` for simplicity since cross-subnet ARP simply >> doesn't happen in our case. >> >> Our IPAM structure: >> >> ``` >> 10.0.0.0/8 (master subnet — VPN points here) >> ├── 10.1.1.0/24 (server 1) >> ├── 10.1.2.0/24 (server 2) >> ├── 10.1.3.0/24 (server 3) >> └── ... (per-server subnets) >> ``` >> >> Each device gets an IP in its server's `/24` child subnet via IPAM >> `request_ip()`. The PoC picks these up automatically — when we add a new >> server and create its child subnet, the next VPN save pushes the updated >> routes. >> >> On our application side, we auto-create IPAM child subnets for new >> servers via the OpenWISP API and sync per-server IPs. All route management >> is handled by the modified OpenWISP code above. >> >> **Before the PoC**, we had a 5-minute cron job re-adding `/24` routes >> directly to the ZeroTier controller API after every OpenWISP-triggered >> wipe. That worked but created a race condition window where routes were >> missing after each sync cycle. The PoC eliminates that entirely. >> >> ### The Limitation: Per-Device Subnet Distribution via VPN Template >> >> The route generation PoC works great for controller-level routing. But >> there's a gap we couldn't solve without modifying OpenWISP further: the VPN >> template still distributes the master subnet to all devices. >> >> Ideally, you'd want device group A to know "you're in 10.1.1.0/24" and >> device group B to know "you're in 10.1.2.0/24" — for example to >> configure the ZT interface mask correctly on each device. But this isn't >> possible today: >> >> 1. **`vpn_subnet` is system-defined and read-only.** The >> `vpn_subnet_{pk}` variable was added in #642/#654 to expose the VPN's >> CIDR to templates. In `vpn.py`, `get_context()` always sets it to the >> VPN's single IPAM subnet: >> ```python >> context[context_keys["vpn_subnet"]] = str(self.subnet.subnet) >> ``` >> The key is `vpn_subnet_{vpn_pk}` — a system-managed variable, not a >> template default. >> >> 2. **VPN context overwrites group variables in the merge order.** In >> `config.py`, the context is built like this: >> ```python >> # Line 958: group variables applied >> context.update(self.device._get_group().get_context()) >> # Line 960: VPN context applied AFTER — overwrites any group values >> with same key >> context.update(self.get_vpn_context()) >> ``` >> Even if you set `vpn_subnet_{pk}` as a group variable, the VPN >> context replaces it two lines later. >> >> 3. **One VPN = one subnet.** A VPN object is tied to a single IPAM >> subnet. There's no mechanism for the VPN template to look up which child >> subnet a device's IP actually belongs to. >> >> So standalone OpenWISP can't distribute per-group or per-device child >> subnets through the VPN template. We work around this in our application >> layer (we know which subnet each device belongs to), but a pure OpenWISP >> deployment would be stuck with the master subnet in the template. >> >> ### Proposal >> >> Two features would make this work natively in OpenWISP: >> >> 1. **Route generation from IPAM child subnets** (the PoC above) — >> opt-in, backward compatible. Could be a boolean on the VPN model like >> `zt_routes_from_children`. Should respect ZeroTier's 128-route limit ( >> `ZT_MAX_NETWORK_ROUTES` in `ZeroTierOne.h`). For deployments needing >> true L2 isolation across subnets, the parent route should support a `via` >> gateway instead of being directly attached — this would prevent >> cross-subnet ARP and fully scope multicast groups to each child subnet. >> >> 2. **Per-device `vpn_child_subnet` variable** — building on the >> `vpn_subnet` variable from #642/#654, when a device's assigned IP falls >> within an IPAM child subnet, expose that child subnet as a new template >> variable. This would let the VPN template configure each device with its >> specific `/24` instead of the master `/8`. >> >> Bonus improvements (could be separate): >> - **Route observability**: Re-enable `routes` in the VPN JSON schema >> (currently stripped in `vpn_backends.py`) and in `_get_repsonse()` — >> right now admins have no visibility into what routes OpenWISP pushes to ZT. >> - **Route merging**: Preserve externally-added routes instead of >> replacing all on every sync — this changes the ownership model though, >> bigger discussion. >> >> ### Context >> >> The ZT backend was implemented during GSoC 2023 (PRs [#778]( >> https://github.com/openwisp/openwisp-controller/pull/778), [#811]( >> https://github.com/openwisp/openwisp-controller/pull/811)) with the >> assumption that one IPAM subnet = one ZT route. The `vpn_subnet` system >> variable was later added in [#642]( >> https://github.com/openwisp/openwisp-controller/issues/642)/[#654]( >> https://github.com/openwisp/openwisp-controller/pull/654) to expose the >> VPN CIDR to templates. Both were reasonable design choices for their scope. >> >> We have this PoC running in production and it's been stable. Happy to >> contribute a proper PR if there's interest. >> >> ### Environment >> - OpenWISP Controller: 1.1.1 >> - OpenWISP IPAM: 0.3.0 >> - ZeroTier: 1.14.x >> >> Best Regards, >> Muhammad. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "OpenWISP" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion, visit >> https://groups.google.com/d/msgid/openwisp/CAKOr0YR%2BY%3D%2B4yTCPE-2NC2Tk%3D0wTV8xhT9uxr%3DcS3tYTFme9Mg%40mail.gmail.com >> <https://groups.google.com/d/msgid/openwisp/CAKOr0YR%2BY%3D%2B4yTCPE-2NC2Tk%3D0wTV8xhT9uxr%3DcS3tYTFme9Mg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "OpenWISP" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion, visit > https://groups.google.com/d/msgid/openwisp/CAAGgX6KB8sS7wa45omi0Y01%2B64CKBizoiKGuw%3DUogo_YHEd9Jw%40mail.gmail.com > <https://groups.google.com/d/msgid/openwisp/CAAGgX6KB8sS7wa45omi0Y01%2B64CKBizoiKGuw%3DUogo_YHEd9Jw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "OpenWISP" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion, visit https://groups.google.com/d/msgid/openwisp/CAKOr0YSN7JeLGhF8rOUr5AiPHY9BbdvZO_ML-aV3v43-%3DeT9eA%40mail.gmail.com.
