Hi Federico Capoano,

Zerotier network carrying mainly Management Traffic and  RADIUS traffic.
I have Openwisp as a controller, and My Portal + FreeRADIUS on another
Server.
Now my customer does not interact with Openwisp. only API do.
So I can customize as much as possible. but i thought this might be an
actual issue.
why not drop everything on the mailing list,
sorry for copy paste I am very lazy..
actually it took some time to reach this final version.

What type should I choose ? Bug or something else?

Regards,
Muhammad Ismail.

On Fri, Apr 17, 2026 at 7:45 PM 'Federico Capoano' via OpenWISP <
[email protected]> wrote:

> Hi Mohammed Ismail,
>
> Can you clarify if you're using Zerotier for management or for
> transporting traffic?
>
> I think it's worth creating an issue, but please do not paste this text as
> is because it's not suited for the issue tracker. Follow the feature
> request template at this URL:
> https://github.com/openwisp/openwisp-controller/issues
>
> Best regards
> *Federico Capoano*
> OpenWISP OÜ
> Kotkapoja tn 2a-10, 10615, Harju maakond, Tallinn, Estonia
> VAT: EE101989729
> +372 59361689
> *openwisp.io* <http://openwisp.io>
>
>
> On Thu, 16 Apr 2026 at 18:55, Mohammed Ismail <[email protected]> wrote:
>
>> ## Subject: ZeroTier route generation from IPAM child subnets — PoC and
>> discussion
>>
>> Hi everyone,
>>
>> We run a multi-tenant WiFi management platform on top of OpenWISP with a
>> growing ZeroTier deployment. We ran into multicast scaling issues with a
>> flat `/8` route and built a working PoC — sharing it here to see if
>> there's interest in upstreaming something like this.
>>
>> ### The Problem
>>
>> The ZeroTier backend currently generates a single route from the VPN's
>> IPAM subnet:
>>
>> ```python
>> config["routes"] = [{"target": str(self.subnet), "via": ""}]
>> ```
>>
>> ZeroTier uses ARP emulation — it converts ARP broadcasts into targeted
>> multicast groups, one per IPv4 address. This works well, but the **Multicast
>> Recipient Limit** (default 32) means that in a large flat network,
>> multicast delivery is capped. With a `/8` route and all members in one
>> broadcast domain, the multicast group for any given IP can't reach all
>> potential peers when the network exceeds the recipient limit. This causes
>> intermittent connectivity issues as ARP responses don't reliably reach all
>> members.
>>
>> Per-server `/24` routes reduce the L2 scope per subnet. With ~250
>> members per `/24` instead of the entire network in one flat `/8`, the
>> multicast groups stay within manageable bounds and ARP resolution works
>> reliably.
>>
>> Additionally:
>> - `vpn_backends.py` removes `routes` from the ZeroTier JSON schema (no
>> UI exposure)
>> - `_get_repsonse()` strips `routes` and `ipAssignmentPools` from ZT API
>> responses (no observability)
>>
>> ### What We Did — PoC on zerotier_service.py
>>
>> We modified `_add_routes_and_ip_assignment()` to generate routes from
>> IPAM child subnets instead of just the parent. Here's the diff:
>>
>> ```diff
>> -        Adds ZeroTier network routes
>> -        and IP assignmentpools through OpenWISP subnet
>> +        Adds ZeroTier network routes from IPAM child subnets (if any)
>> +        and IP assignment pools through OpenWISP subnet.
>> +
>> +        If the VPN's IPAM subnet has child subnets, generates routes for
>> +        both the parent (cross-subnet routing) and each child (ARP
>> isolation).
>> +        Otherwise falls back to single route from parent subnet.
>>
>> -        config["routes"] = [{"target": str(self.subnet), "via": ""}]
>> +        try:
>> +            from openwisp_ipam.models import Subnet
>> +            parent =
>> Subnet.objects.filter(subnet=str(self.subnet)).first()
>> +            if parent:
>> +                children = list(parent.child_subnet_set.only('subnet'))
>> +                if children:
>> +                    # Parent route ensures cross-subnet reachability
>> +                    # Child routes (more specific) provide ARP isolation
>> +                    config["routes"] = [
>> +                        {"target": str(self.subnet), "via": ""}
>> +                    ] + [
>> +                        {"target": str(child.subnet), "via": ""}
>> +                        for child in children
>> +                    ]
>> +                else:
>> +                    config["routes"] = [{"target": str(self.subnet),
>> "via": ""}]
>> +            else:
>> +                config["routes"] = [{"target": str(self.subnet), "via":
>> ""}]
>> +        except Exception:
>> +            # Fallback: original behavior if anything fails
>> +            config["routes"] = [{"target": str(self.subnet), "via": ""}]
>> ```
>>
>> This gives us a parent `/8` route (cross-subnet reachability) plus
>> per-server `/24` child routes.
>>
>> **Important caveat about L2 isolation**: The parent `/8` route with `via:
>> ""` (directly attached) means devices still consider the entire `/8` as
>> L2-local. For cross-subnet traffic, devices would still ARP through the
>> flat `/8`, hitting the multicast recipient limit. In our deployment this
>> doesn't matter — our traffic is entirely intra-/24 (hotspots only talk to
>> their server's RADIUS at `.1` and controller at `.254`). But for
>> deployments that need cross-subnet communication, the parent route should
>> use a gateway (`"via": "<router_zt_ip>"`) instead of being directly
>> attached. We kept `via: ""` for simplicity since cross-subnet ARP simply
>> doesn't happen in our case.
>>
>> Our IPAM structure:
>>
>> ```
>> 10.0.0.0/8 (master subnet — VPN points here)
>> ├── 10.1.1.0/24 (server 1)
>> ├── 10.1.2.0/24 (server 2)
>> ├── 10.1.3.0/24 (server 3)
>> └── ... (per-server subnets)
>> ```
>>
>> Each device gets an IP in its server's `/24` child subnet via IPAM
>> `request_ip()`. The PoC picks these up automatically — when we add a new
>> server and create its child subnet, the next VPN save pushes the updated
>> routes.
>>
>> On our application side, we auto-create IPAM child subnets for new
>> servers via the OpenWISP API and sync per-server IPs. All route management
>> is handled by the modified OpenWISP code above.
>>
>> **Before the PoC**, we had a 5-minute cron job re-adding `/24` routes
>> directly to the ZeroTier controller API after every OpenWISP-triggered
>> wipe. That worked but created a race condition window where routes were
>> missing after each sync cycle. The PoC eliminates that entirely.
>>
>> ### The Limitation: Per-Device Subnet Distribution via VPN Template
>>
>> The route generation PoC works great for controller-level routing. But
>> there's a gap we couldn't solve without modifying OpenWISP further: the VPN
>> template still distributes the master subnet to all devices.
>>
>> Ideally, you'd want device group A to know "you're in 10.1.1.0/24" and
>> device group B to know "you're in 10.1.2.0/24" — for example to
>> configure the ZT interface mask correctly on each device. But this isn't
>> possible today:
>>
>> 1. **`vpn_subnet` is system-defined and read-only.** The
>> `vpn_subnet_{pk}` variable was added in #642/#654 to expose the VPN's
>> CIDR to templates. In `vpn.py`, `get_context()` always sets it to the
>> VPN's single IPAM subnet:
>>    ```python
>>    context[context_keys["vpn_subnet"]] = str(self.subnet.subnet)
>>    ```
>>    The key is `vpn_subnet_{vpn_pk}` — a system-managed variable, not a
>> template default.
>>
>> 2. **VPN context overwrites group variables in the merge order.** In
>> `config.py`, the context is built like this:
>>    ```python
>>    # Line 958: group variables applied
>>    context.update(self.device._get_group().get_context())
>>    # Line 960: VPN context applied AFTER — overwrites any group values
>> with same key
>>    context.update(self.get_vpn_context())
>>    ```
>>    Even if you set `vpn_subnet_{pk}` as a group variable, the VPN
>> context replaces it two lines later.
>>
>> 3. **One VPN = one subnet.** A VPN object is tied to a single IPAM
>> subnet. There's no mechanism for the VPN template to look up which child
>> subnet a device's IP actually belongs to.
>>
>> So standalone OpenWISP can't distribute per-group or per-device child
>> subnets through the VPN template. We work around this in our application
>> layer (we know which subnet each device belongs to), but a pure OpenWISP
>> deployment would be stuck with the master subnet in the template.
>>
>> ### Proposal
>>
>> Two features would make this work natively in OpenWISP:
>>
>> 1. **Route generation from IPAM child subnets** (the PoC above) —
>> opt-in, backward compatible. Could be a boolean on the VPN model like
>> `zt_routes_from_children`. Should respect ZeroTier's 128-route limit (
>> `ZT_MAX_NETWORK_ROUTES` in `ZeroTierOne.h`). For deployments needing
>> true L2 isolation across subnets, the parent route should support a `via`
>> gateway instead of being directly attached — this would prevent
>> cross-subnet ARP and fully scope multicast groups to each child subnet.
>>
>> 2. **Per-device `vpn_child_subnet` variable** — building on the
>> `vpn_subnet` variable from #642/#654, when a device's assigned IP falls
>> within an IPAM child subnet, expose that child subnet as a new template
>> variable. This would let the VPN template configure each device with its
>> specific `/24` instead of the master `/8`.
>>
>> Bonus improvements (could be separate):
>> - **Route observability**: Re-enable `routes` in the VPN JSON schema
>> (currently stripped in `vpn_backends.py`) and in `_get_repsonse()` —
>> right now admins have no visibility into what routes OpenWISP pushes to ZT.
>> - **Route merging**: Preserve externally-added routes instead of
>> replacing all on every sync — this changes the ownership model though,
>> bigger discussion.
>>
>> ### Context
>>
>> The ZT backend was implemented during GSoC 2023 (PRs [#778](
>> https://github.com/openwisp/openwisp-controller/pull/778), [#811](
>> https://github.com/openwisp/openwisp-controller/pull/811)) with the
>> assumption that one IPAM subnet = one ZT route. The `vpn_subnet` system
>> variable was later added in [#642](
>> https://github.com/openwisp/openwisp-controller/issues/642)/[#654](
>> https://github.com/openwisp/openwisp-controller/pull/654) to expose the
>> VPN CIDR to templates. Both were reasonable design choices for their scope.
>>
>> We have this PoC running in production and it's been stable. Happy to
>> contribute a proper PR if there's interest.
>>
>> ### Environment
>> - OpenWISP Controller: 1.1.1
>> - OpenWISP IPAM: 0.3.0
>> - ZeroTier: 1.14.x
>>
>> Best Regards,
>> Muhammad.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "OpenWISP" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To view this discussion, visit
>> https://groups.google.com/d/msgid/openwisp/CAKOr0YR%2BY%3D%2B4yTCPE-2NC2Tk%3D0wTV8xhT9uxr%3DcS3tYTFme9Mg%40mail.gmail.com
>> <https://groups.google.com/d/msgid/openwisp/CAKOr0YR%2BY%3D%2B4yTCPE-2NC2Tk%3D0wTV8xhT9uxr%3DcS3tYTFme9Mg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "OpenWISP" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion, visit
> https://groups.google.com/d/msgid/openwisp/CAAGgX6KB8sS7wa45omi0Y01%2B64CKBizoiKGuw%3DUogo_YHEd9Jw%40mail.gmail.com
> <https://groups.google.com/d/msgid/openwisp/CAAGgX6KB8sS7wa45omi0Y01%2B64CKBizoiKGuw%3DUogo_YHEd9Jw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"OpenWISP" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion, visit 
https://groups.google.com/d/msgid/openwisp/CAKOr0YSN7JeLGhF8rOUr5AiPHY9BbdvZO_ML-aV3v43-%3DeT9eA%40mail.gmail.com.

Reply via email to