--- Begin Message ---
Hi,
Entry in deployment.yml as follows:
- tag: 'FORTINET'
driver: cli
platform: FortiOS
only:
- 10.10.10.53/32
username: testloginacct
password: redacted
ssh_master_opts:
- "-o"
- "StrictHostKeyChecking=no"
See output below ; it looks like the script detects 2 VDOMS, but failure
afterwards…
ND2_LOG_PLUGINS=1 netdisco-do arpnip -DQ -d 10.10.10.53
[2888572] 2025-03-18 20:35:48 info App::Netdisco version 2.076005 loaded.
[2888572] 2025-03-18 20:35:48 info arpnip: [10.10.10.53] started at Tue Mar 18
13:35:48 2025
SELECT me.version, me.installed
FROM dbix_class_schema_versions me
WHERE 1 = 0
SELECT me.version
FROM dbix_class_schema_versions me
ORDER BY installed DESC
LIMIT '1'
SELECT me.ip, me.alias, me.subnet, me.port, me.dns, me.creation
FROM device_ip me
WHERE me.alias = '10.10.10.53' AND me.ip = '10.10.10.53'
SELECT me.ip, me.creation, me.dns, me.description, me.uptime, me.contact,
me.name, me.location, me.layers, me.num_ports, me.mac, me.serial,
me.chassis_id, me.model, me.ps1_type, me.ps2_type, me.ps1_status,
me.ps2_status, me.fan, me.slots, me.vendor, me.os, me.os_ver, me.log,
me.snmp_ver, me.snmp_comm, me.snmp_class, me.snmp_engineid, me.vtp_domain,
me.vtp_mode, me.last_discover, me.last_macsuck, me.last_arpnip, me.is_pseudo,
me.pae_is_enabled, me.custom_fields, me.tags, to_char( me.creation, 'YYYY-MM-DD
HH24:MI' ), to_char( me.last_arpnip, 'YYYY-MM-DD HH24:MI' ), to_char(
me.last_discover, 'YYYY-MM-DD HH24:MI' ), to_char( me.last_macsuck, 'YYYY-MM-DD
HH24:MI' ), extract( epoch
FROM age( LOCALTIMESTAMP, me.creation ) ), extract( epoch
FROM age( LOCALTIMESTAMP, me.last_arpnip ) ), extract( epoch
FROM age( LOCALTIMESTAMP, me.last_discover ) ), extract( epoch
FROM age( LOCALTIMESTAMP, me.last_macsuck ) ), replace( age( timestamp
'epoch' + me.uptime / 100 * interval '1 second', timestamp '1970-01-01
00:00:00-00' ) ::text, 'mon', 'month' )
FROM device me
WHERE me.ip = '10.10.10.53'
[2888572] 2025-03-18 20:35:48 debug loading worker plugin
App::Netdisco::Worker::Plugin::Internal::BackendFQDN
[2888572] 2025-03-18 20:35:48 debug loading worker plugin
App::Netdisco::Worker::Plugin::Internal::SNMPFastDiscover
[2888572] 2025-03-18 20:35:48 debug loading worker plugin
App::Netdisco::Worker::Plugin::Arpnip
[2888572] 2025-03-18 20:35:48 debug loading worker plugin
App::Netdisco::Worker::Plugin::Arpnip::Hooks
[2888572] 2025-03-18 20:35:48 debug loading worker plugin
App::Netdisco::Worker::Plugin::Arpnip::Nodes
[2888572] 2025-03-18 20:35:48 debug loading worker plugin
App::Netdisco::Worker::Plugin::Arpnip::Subnets
[2888572] 2025-03-18 20:35:48 debug arpnip: running with timeout 600s
[2888572] 2025-03-18 20:35:48 debug //// CHECK \\\\ phase
[2888572] 2025-03-18 20:35:48 debug ⮕ worker Internal::BackendFQDN p1000000
[2888572] 2025-03-18 20:35:48 debug ⮕ worker Internal::SNMPFastDiscover p1000000
[2888572] 2025-03-18 20:35:48 debug running with configured SNMP timeouts
[2888572] 2025-03-18 20:35:48 debug ⮕ worker Arpnip p0
[2888572] 2025-03-18 20:35:48 debug ⬅ (done) arpnip is able to run
[2888572] 2025-03-18 20:35:48 debug //// EARLY \\\\ phase
[2888572] 2025-03-18 20:35:48 debug ⮕ worker Arpnip::Nodes p0 "prepare common
data"
[2888572] 2025-03-18 20:35:48 debug //// MAIN \\\\ phase
[2888572] 2025-03-18 20:35:48 debug ⮕ worker Arpnip::Nodes p1000000
[2888572] 2025-03-18 20:35:48 debug ⬅ (info) skip: arp table data supplied by
other source
[2888572] 2025-03-18 20:35:48 debug ⮕ worker Arpnip::Nodes p200
[2888572] 2025-03-18 20:35:48 debug cli session cache warm: [10.10.10.53]
[2888572] 2025-03-18 20:35:49 debug 10.10.10.53 2888572 arpnip()
[2888572] 2025-03-18 20:35:50 debug skipping through --More-- pagination
[2888572] 2025-03-18 20:35:50 debug output collected: get system status
[2888572] 2025-03-18 20:35:50 debug output collected: Version: FortiGate-600E
v7.0.8,build0418,221012 (GA.F)
[2888572] 2025-03-18 20:35:50 debug output collected: Firmware Signature:
certified
[2888572] 2025-03-18 20:35:50 debug output collected: Virus-DB:
93.01781(2025-03-18 11:26)
[2888572] 2025-03-18 20:35:50 debug output collected: Extended DB:
93.01781(2025-03-18 11:25)
[2888572] 2025-03-18 20:35:50 debug output collected: Extreme DB:
1.00000(2018-04-09 18:07)
[2888572] 2025-03-18 20:35:50 debug output collected: AV AI/ML Model:
4.00906(2025-03-18 11:45)
[2888572] 2025-03-18 20:35:50 debug output collected: IPS-DB:
6.00741(2015-12-01 02:30)
[2888572] 2025-03-18 20:35:50 debug output collected: IPS-ETDB:
31.00972(2025-03-18 00:07)
[2888572] 2025-03-18 20:35:50 debug output collected: APP-DB:
31.00972(2025-03-18 00:06)
[2888572] 2025-03-18 20:35:50 debug output collected: INDUSTRIAL-DB:
18.00193(2021-11-09 02:09)
[2888572] 2025-03-18 20:35:50 debug output collected: IPS Malicious URL
Database: 5.00356(2025-03-18 05:13)
[2888572] 2025-03-18 20:35:50 debug output collected: Serial-Number: REDACTED
[2888572] 2025-03-18 20:35:50 debug output collected: BIOS version: 05000006
[2888572] 2025-03-18 20:35:50 debug output collected: System Part-Number:
P24088-03
[2888572] 2025-03-18 20:35:50 debug output collected: Log hard disk: Not
available
[2888572] 2025-03-18 20:35:50 debug output collected: Hostname: Fortinet-UUT-FW
[2888572] 2025-03-18 20:35:50 debug output collected: Private Encryption:
Disable
[2888572] 2025-03-18 20:35:50 debug output collected: Operation Mode: NAT
[2888572] 2025-03-18 20:35:50 debug output collected: Current virtual domain:
FG-traffic
[2888572] 2025-03-18 20:35:50 debug output collected: Max number of virtual
domains: 10
[2888572] 2025-03-18 20:35:50 debug output collected: Virtual domains status: 2
in NAT mode, 0 in TP mode
[2888572] 2025-03-18 20:35:50 debug output collected: Virtual domain
configuration: split-task
[2888572] 2025-03-18 20:35:50 debug output collected: FIPS-CC mode: disable
[2888572] 2025-03-18 20:35:50 debug output collected: Current HA mode: a-p,
primary
[2888572] 2025-03-18 20:35:50 debug output collected: Cluster uptime: 1221
days, 20 hours, 22 minutes, 58 seconds
[2888572] 2025-03-18 20:35:50 debug output collected: Cluster state change
time: 2022-11-26 20:06:54
[2888572] 2025-03-18 20:35:50 debug output collected: Branch point: 0418
[2888572] 2025-03-18 20:35:50 debug output collected: Release Version
Information: GA
[2888572] 2025-03-18 20:35:50 debug output collected: FortiOS x86-64: Yes
[2888572] 2025-03-18 20:35:50 debug output collected: System time: Wed Mar 19
04:35:50 2025
[2888572] 2025-03-18 20:35:50 debug output collected: Last reboot reason: warm
reboot
[2888572] 2025-03-18 20:35:50 debug output collected: Fortinet-UUT-FW
[2888572] 2025-03-18 20:35:50 debug output collected: get system arp
[2888572] 2025-03-18 20:35:50 debug output collected: command parse error
before 'arp'
[2888572] 2025-03-18 20:35:50 debug output collected: Command fail. Return code
-61
[2888572] 2025-03-18 20:35:50 debug output collected: Fortinet-UUT-FW
[2888572] 2025-03-18 20:35:51 debug output collected: diagnose ipv6
neighbor-cache list
[2888572] 2025-03-18 20:35:51 debug output collected: 8497: Unknown action 0
[2888572] 2025-03-18 20:35:51 debug output collected: Command fail. Return code
-1
[2888572] 2025-03-18 20:35:51 debug output collected: Fortinet-UUT-FW
[2888572] 2025-03-18 20:35:51 debug ⬅ (done) Gathered arp caches from
10.10.10.53
[2888572] 2025-03-18 20:35:51 debug ⮕ worker Arpnip::Nodes p100
[2888572] 2025-03-18 20:35:51 debug ⬅ (info) skip: namespace passed at higher
priority
[2888572] 2025-03-18 20:35:51 debug ⮕ worker Arpnip::Subnets p100
[2888572] 2025-03-18 20:35:51 debug snmp reader cache warm: [10.10.10.53]
SELECT me.ip, me.snmp_comm_rw, me.snmp_auth_tag_read, me.snmp_auth_tag_write
FROM community me
WHERE me.ip = '10.10.10.53'
[2888572] 2025-03-18 20:35:51 debug [10.10.10.53:161] try_connect with v: 2, t:
0.2, r: 0, class: SNMP::Info::Layer3::Fortinet, comm: <hidden>
[2888572] 2025-03-18 20:35:52 debug [10.10.10.53] arpnip - found subnet
10.10.128.0/25
[2888572] 2025-03-18 20:35:52 debug [10.10.10.53] arpnip - found subnet
10.10.129.0/24
[2888572] 2025-03-18 20:35:52 debug [10.10.10.53] arpnip - found subnet
10.10.188.0/24
[2888572] 2025-03-18 20:35:52 debug [10.10.10.53] arpnip - found subnet
10.10.140.0/24
[2888572] 2025-03-18 20:35:52 debug [10.10.10.53] arpnip - found subnet
10.10.10..0/24
[2888572] 2025-03-18 20:35:52 debug [10.10.10.53] arpnip - found subnet
192.168.10.0/24
[2888572] 2025-03-18 20:35:52 debug [10.10.10.53] arpnip - found subnet
169.254.1.0/24
[2888572] 2025-03-18 20:35:52 debug [10.10.10.53] arpnip - found subnet
10.10.130.0/24
[2888572] 2025-03-18 20:35:52 debug [10.10.10.53] arpnip - found subnet
10.10.128.128/25
BEGIN WORK
SELECT me.net, me.creation, me.last_discover
FROM subnets me
WHERE me.net = '10.10.128.0/25' FOR UPDATE
UPDATE subnets
SET last_discover = to_timestamp( 1742330152.489542 ) ::timestamp
WHERE net = '10.10.128.0/25'
COMMIT
BEGIN WORK
SELECT me.net, me.creation, me.last_discover
FROM subnets me
WHERE me.net = '10.10.129.0/24' FOR UPDATE
UPDATE subnets
SET last_discover = to_timestamp( 1742330152.489542 ) ::timestamp
WHERE net = '10.10.129.0/24'
COMMIT
BEGIN WORK
SELECT me.net, me.creation, me.last_discover
FROM subnets me
WHERE me.net = '10.10.188.0/24' FOR UPDATE
UPDATE subnets
SET last_discover = to_timestamp( 1742330152.489542 ) ::timestamp
WHERE net = '10.10.188.0/24'
COMMIT
BEGIN WORK
SELECT me.net, me.creation, me.last_discover
FROM subnets me
WHERE me.net = '10.10.140.0/24' FOR UPDATE
UPDATE subnets
SET last_discover = to_timestamp( 1742330152.489542 ) ::timestamp
WHERE net = '10.10.140.0/24'
COMMIT
BEGIN WORK
SELECT me.net, me.creation, me.last_discover
FROM subnets me
WHERE me.net = '10.10.10..0/24' FOR UPDATE
UPDATE subnets
SET last_discover = to_timestamp( 1742330152.489542 ) ::timestamp
WHERE net = '10.10.10..0/24'
COMMIT
BEGIN WORK
SELECT me.net, me.creation, me.last_discover
FROM subnets me
WHERE me.net = '192.168.10.0/24' FOR UPDATE
UPDATE subnets
SET last_discover = to_timestamp( 1742330152.489542 ) ::timestamp
WHERE net = '192.168.10.0/24'
COMMIT
BEGIN WORK
SELECT me.net, me.creation, me.last_discover
FROM subnets me
WHERE me.net = '169.254.1.0/24' FOR UPDATE
UPDATE subnets
SET last_discover = to_timestamp( 1742330152.489542 ) ::timestamp
WHERE net = '169.254.1.0/24'
COMMIT
BEGIN WORK
SELECT me.net, me.creation, me.last_discover
FROM subnets me
WHERE me.net = '10.10.130.0/24' FOR UPDATE
UPDATE subnets
SET last_discover = to_timestamp( 1742330152.489542 ) ::timestamp
WHERE net = '10.10.130.0/24'
COMMIT
BEGIN WORK
SELECT me.net, me.creation, me.last_discover
FROM subnets me
WHERE me.net = '10.10.128.128/25' FOR UPDATE
UPDATE subnets
SET last_discover = to_timestamp( 1742330152.489542 ) ::timestamp
WHERE net = '10.10.128.128/25'
COMMIT
[2888572] 2025-03-18 20:35:52 debug ⬅ (info) [10.10.10.53] arpnip - processed
9 Subnet entries
[2888572] 2025-03-18 20:35:52 debug //// STORE \\\\ phase
[2888572] 2025-03-18 20:35:52 debug ⮕ worker Arpnip::Nodes p0
[2888572] 2025-03-18 20:35:52 debug resolving 0 ARP entries with max 50
outstanding requests
[2888572] 2025-03-18 20:35:52 debug [10.10.10.53] arpnip - processed 0 ARP
Cache entries
[2888572] 2025-03-18 20:35:52 debug [10.10.10.53] arpnip - processed 0 IPv6
Neighbor Cache entries
UPDATE device
SET last_arpnip = to_timestamp( 1742330148.406285 ) ::timestamp
WHERE ip = '10.10.10.53'
[2888572] 2025-03-18 20:35:52 debug ⬅ (done) Ended arpnip for 10.10.10.53
[2888572] 2025-03-18 20:35:52 debug //// LATE \\\\ phase
[2888572] 2025-03-18 20:35:52 debug ⮕ worker Arpnip::Hooks p0
[2888572] 2025-03-18 20:35:52 debug ⬅ (info) [10.10.10.53] hooks - 0 queued
[2888572] 2025-03-18 20:35:52 info arpnip: finished at Tue Mar 18 13:35:52 2025
[2888572] 2025-03-18 20:35:52 info arpnip: status done: Ended arpnip for
10.10.10.53
From: Christian Ramseyer <ramse...@netnea.com>
Sent: Tuesday, March 18, 2025 1:01 PM
To: Christian Vo <christian...@synaptics.com>; Michael Butash
<mich...@butash.net>
Cc: netdisco-users@lists.sourceforge.net
Subject: Re: [Netdisco] Fortinet FortiOS Shenanigans
CAUTION: Email originated externally, do not click links or open attachments
unless you recognize the sender and know the content is safe.
On 18.03.2025 19:55, Christian Vo wrote:
Hi all,
I’m running into Fortinet-related arpnip issue, it seems that the CLI “get
system arp” cmd need to be executed from a specific VDOM.
Initial SSH login from the account specified in the deployment.yml will fail to
run the above cmd.
I noticed we needed to do the following from CLI:
1. config vdom
2. edit FG-traffic (not sure if this is default VDOM name or not, I do see
root as the other option)
3. get system arp
I do realize netdisco-sshcollector is depreciated, so not entire sure what is
needed on my end in order for these cmds to be ran properly.
Hi
Just the standalone netdisco-sshcollector script that used to come with
netdisco is deprecated, if you configure device_auth like discussed in this
thread earlier, e.g.
device_auth:
- tag: 'corp_fortinet'
#action: arpnip::nodes
#only: 'group:Fortinet-Fortigate'
only:
- 192.168.1.42
driver: cli
platform: FortiOS
username: 'monitor-user'
password: 'secret'
banner: true
ssh_master_opts:
- "-o"
- "StrictHostKeyChecking=no"
this will use
https://github.com/netdisco/netdisco/blob/master/lib/App/Netdisco/SSHCollector/Platform/FortiOS.pm<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_netdisco_netdisco_blob_master_lib_App_Netdisco_SSHCollector_Platform_FortiOS.pm&d=DwMDaQ&c=7dfBJ8cXbWjhc0BhImu8wVIoUFmBzj1s88r8EGyM0UY&r=o6o4SQ5I8KCNLqn8RR0gVWq0M-4VyiDyieVGPvZlceI&m=8fcVlN6rHrZMHa-MYQnliKSk4T2YozFyt-CDxjLoc05PhTyQXe_PGe1k6XmuE2dV&s=61l86i0OJkjQedQTWYHNWHbJ85i66BEFxLPjeDNPvo8&e=>
inside the regular Netdisco arpnip process. The latest version of the module
tries to enumerate all vdoms and then get v4/v6 table for all of them.
To see if it works, try
ND2_LOG_PLUGINS=1 netdisco-do arpnip -DQ -d <ip>
and optimally you'll see some SSH-related output in there.
Cheers
Christian
Please help
Christian
From: Michael Butash <mich...@butash.net><mailto:mich...@butash.net>
Sent: Sunday, March 16, 2025 4:23 PM
To: Christian Ramseyer <ramse...@netnea.com><mailto:ramse...@netnea.com>
Cc:
netdisco-users@lists.sourceforge.net<mailto:netdisco-users@lists.sourceforge.net>
Subject: Re: [Netdisco] Fortinet FortiOS Shenanigans
CAUTION: Email originated externally, do not click links or open attachments
unless you recognize the sender and know the content is safe.
Ahh, yes chmod 755 on the directory worked for the arpnip! Ok, I guess I'm not
a very good sysadmin not knowing/realizing that as a security thing, but didn't
think (and never have) much about it.
Great, I was going more for completeness since I saw there was a forti ssh
collector now in there, as neighbors aren't figuring themselves out for me at
all here. LLDP still isn't figuring out neighbors between my catalysts and
fortiswitches, but it's a small enough network I just added manual links for
them anyways. I may still follow up separately on that as LLDP info is being
found, just not showing on ports or linking topology neighbors.
Regarding having to bulkwalk_no the single host, I'd probably blame Fortinet if
it weren't for the fact 2 out of the 3 work normally, another hub and a branch
spoke all poll just fine. Even weirder it gets stuck on a stack oid that most
certainly isn't present on the fortigate, and repeatedly with no delay or
waiting for a response. The fortigate seems to ignore it as an invalid mib, but
they show incoming as quick as the terminal will scroll.
I'm curious enough now I'm going to pull a full debug on the working and
not-working devices (both virtual in azure too, the spoke is a physical 100F)
and see, maybe open a ticket too as I've had enough weirdness on this
deployment we're old friends they and I. I'll follow up if I get a better
answer. I'm open to share them unicast back to you if interested to have a look.
Oh, separate note on your memory leak - maybe unrelated, but thought I'd
mention it... Another long-time customer of mine randomly started having
fortigate issues back in October with the IPS randomly starting to OOM the box
into conservation mode, and after a few months found out it was a bad IPS
engine update they pushed randomly (oct 26th if I remember right). I think it
got fixed officially in a january maintenance release, but I had to get a
specifically fixed IPS engine version for some older 7.2 boxes we had that we
didn't want to upgrade yet. IPS stuck out like honeymoon wood for the top
memory consumer out of nowhere on only one box of many. I was surprised that it
didn't get more public attention affecting folks, fortinet support seemed to
know about it after a few cases over months about it running through hoops.
Thanks again Christian, really appreciate the answers and your experience!
-mb
On Sun, Mar 16, 2025 at 2:42 PM Christian Ramseyer
<ramse...@netnea.com<mailto:ramse...@netnea.com>> wrote:
On 16.03.2025 22:12, Michael Butash wrote:
> Ahh, so money! Yeah, once I found a reference on how to set bulkget_no
> (thanks mailing list, your docs *should* really give an example of
> use.. :)), it ran right through with no issues using getnext. Reading
> the doc didn't make any sense where to use it, searches turn up
> nothing on how or where to declare this, even chatgpt said I should
> jab it into the device_auth section, but otherwise... Thank you!!
>
> So now the question is why is ND misbehaving? There really is little
> configuration difference between the working fortigate and not working
> one, particularly nothing special around SNMP, so I have no idea why
> ND would behave like this for one fortigate and not another. This
> seems more of a ND problem than the fortigate.
Nice we've made some progress, excellent :)
I doubt its ND, having crappy bulkget implementations is a tradition
across many vendors. I'm pretty sure you'll get the same result when
pointing "snmpbulkwalk" at the same ifStackStatus subtree.
>
> And yes, I'm a dork re: discover vs arpnip, I was doing discovery.
> Sorry for barking up the wrong tree.
>
> Still though, it seems to try, but fails weirdly with an error about
> .libnet-openssh-perl not being secure. I wasn't really sure what part
> it was considering "not secure", chatgpt seemed to think it was
> related to the directory not being secure, but it's chmod 700 to
> netdisco only, not sure how much more secure it wants it. I can ssh to
> the device normally with that account otherwise from the server.
>
> [2179658] 2025-03-16 20:41:55 error [10.0.0.10] ssh connection error
> [ctl_dir /opt/netdisco/.libnet-openssh-perl/ is not secure]
> [2179658] 2025-03-16 20:41:55 debug ⬅ (defer) arpnip failed: could not
> SSH connect to 10.0.0.10
>
It's probably the /opt/netdisco directory that's still group writeable
or worse, openssh doesn't like that. chmod 755 (or 700, 750) on it
should fix it.
Cheers
Christian
--
Christian Ramseyer, netnea ag
Network Management. Security. OpenSource.
https://www.netnea.com<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.netnea.com&d=DwMDaQ&c=7dfBJ8cXbWjhc0BhImu8wVIoUFmBzj1s88r8EGyM0UY&r=o6o4SQ5I8KCNLqn8RR0gVWq0M-4VyiDyieVGPvZlceI&m=8fcVlN6rHrZMHa-MYQnliKSk4T2YozFyt-CDxjLoc05PhTyQXe_PGe1k6XmuE2dV&s=Bg_e03yJ9ZtnFWwfuCLM_B_5WAi4C3jE8Jb2WwZVNE8&e=>
Phone: +41 79 644 77 64
--- End Message ---
