Hi everyone,

I would like to share a small OpenWISP add-on I have just published: openwisp-netconsole.

  https://github.com/mikysal78/openwisp-netconsole

WHAT IT DOES

It adds a "Netconsole" tab to the device page of the OpenWISP admin. The tab shows, in real time, the kernel messages that the router sends with Linux netconsole and that rsyslog stores on the server. It refreshes by itself, asks the server only for the new lines and lets you download the rotated log files.

It also adds a "Router log rotation" page (superusers only) to choose how often the router logs are rotated, their maximum size and how long they are kept, without editing configuration files on the server.

The repository also contains:
- an OpenWISP template for the devices: a hotplug script that loads netconsole and tags every message with the device name; - a basic Ansible playbook for the official ansible-openwisp2 role, with the rsyslog configuration and the rotation script.

HOW IT WAS BORN

I run the OpenWISP instance of NINUX Basilicata, a community network in southern Italy: a handful of OpenWrt routers (TP-Link, Linksys, TOTOLINK, Zyxel and an x86 border router) connected by an 802.11s mesh, cables and ZeroTier, with Babel routing on BIRD.

Some of these routers occasionally hung, and the remote syslog was useless:
when a router freezes or panics, syslog has no time to send anything. So we started sending the kernel messages with netconsole, which works until the very last moment, and collecting them with rsyslog.

That raised two problems:
1. netconsole has no hostname: the logs were stored by source IP, and a router changes IP when it moves between LAN, WAN, mesh and VPN, so the history of one router ended up in several folders; 2. to read them you had to log into the server, while everything else about the device was already in OpenWISP.

For the first one, the router now adds the device name as netconsole "userdata" (extended format), so rsyslog keeps one folder per device. This needs a kernel with CONFIG_NETCONSOLE_DYNAMIC, which OpenWrt builds as "n", so we enable it in our firmware build. For the second one, this module shows the logs where you already look: on the device page.

WHY A SEPARATE PACKAGE

It is a small Django app that extends the device admin already registered (including the openwisp-monitoring one) instead of replacing it, so it can be added to an existing instance with two settings (openwisp2_extra_python_packages and openwisp2_extra_django_apps).

Since it reads files on the server and the rotation is done by root, I paid particular attention to security: - only users who can see the device can read its logs, with the usual   organization filtering; - the browser never sends a path: it can only choose among the folders computed by the server, symlinks are rejected and files are opened with O_NOFOLLOW; - a folder whose name or IP is shared with another device is not shown, so logs cannot leak between devices or organizations;
- log text is inserted into the page as plain text only;
- OpenWISP never writes system configuration: the rotation page only saves a small JSON file that a root script validates again (strict ranges, fixed template) before generating the logrotate configuration.

The repository has CI (tests on Python 3.10-3.13, ruff, checks of the example playbook and of the router script, rsyslog validation), CodeQL, Dependabot, a security policy for private vulnerability reporting, and releases with a built wheel:

  pip install https://github.com/mikysal78/openwisp-netconsole/releases/download/v0.1.1/openwisp_netconsole-0.1.1-py3-none-any.whl

It is still a 0.1 and it was born for our needs, so feedback, issues and pull requests are very welcome. If you think something like this could be useful in OpenWISP itself, I would be glad to discuss it.

Best regards,
Michele Salerno

--
You received this message because you are subscribed to the Google Groups 
"OpenWISP" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion, visit 
https://groups.google.com/d/msgid/openwisp/b536a73c-c3b6-4578-b9c0-e40bf6e027da%40gmail.com.

Reply via email to