Send Netdot-devel mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://osl.uoregon.edu/mailman/listinfo/netdot-devel
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Netdot-devel digest..."
Today's Topics:
1. [Netdot - Bug #1818] (Resolved) Could not insert base MAC: :
Error while inserting PhysAddr: Missing address
([email protected])
2. [Netdot - Feature #1824] (Closed) Improve Depth Up / Depth
Down display ([email protected])
3. [SCM] Netdot branch master updated. netdot-1.0.6-24-g2c3925c
([email protected])
4. [SCM] Netdot branch netdot-1.0 updated.
netdot-1.0.6-24-g2c3925c ([email protected])
5. [Netdot - Feature #1791] (Resolved) Exclude static IP
information from prune_db.pl's IP address pruning
([email protected])
6. [Netdot - Bug #1818] Could not insert base MAC: : Error while
inserting PhysAddr: Missing address ([email protected])
----------------------------------------------------------------------
Message: 1
Date: Fri, 27 Jun 2014 10:33:18 -0700
From: [email protected]
Subject: [Netdot-devel] [Netdot - Bug #1818] (Resolved) Could not
insert base MAC: : Error while inserting PhysAddr: Missing address
To: [email protected], [email protected]
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
Issue #1818 has been updated by Carlos Vicente.
Category set to DeviceManagement
Status changed from New to Resolved
Assignee set to Brian Candler
Target version set to 1.0.7
Resolution set to fixed
Problem 1:
Very strange because notice that the physaddr value passed in the SQL insert
statement is "2". I re-read the code and could not figure out how that may have
happened. I did change the way that the parameter is assigned to the Asset
insert code to not assign the key to the hash unless the return value is
non-null.
Problem2:
The reason for excluding "me|bme" interfaces was that Junos uses them for the
control plane, assigning them RFC-1918 addresses, which cause bogus subnets to
appear in the address space. I have removed those two patterns from the
default, but added some comments recommending it to Junos users.
Closing this for now, but feel free to reopen if the issue reappears.
----------------------------------------
Bug #1818: Could not insert base MAC: : Error while inserting PhysAddr: Missing
address
https://osl.uoregon.edu/redmine/issues/1818#change-3207
Author: Brian Candler
Status: Resolved
Priority: Normal
Assignee: Brian Candler
Category: DeviceManagement
Target version: 1.0.7
Resolution: fixed
[Netdot 1.0.6, configured with IFNAME_SHORT => 1, running under Ubuntu 14.04]
This is when adding the first device, which is the Netdot server itself,
running snmpd.
It was added via the GUI and using SNMP, and then I ran from the CLI:
<pre>
/usr/local/netdot/bin/updatedevices.pl -DIFAT -d
...
DEBUG - Device::get_snmp_info: Finished getting SNMP info from x.x.x.21
DEBUG - monitor.example.net did not return base MAC. Using first available
interface MAC.
Use of uninitialized value $address in sprintf at
/usr/local/netdot/lib/Netdot/Model/Device.pm line 5641.
DEBUG - monitor.example.net: Could not insert base MAC: : Error while inserting
PhysAddr: Missing address
DEBUG - monitor.example.net: No suitable base MAC found
ERROR - Error while inserting Asset: Can't insert new Asset: DBD::mysql::st
execute failed: Cannot add or update a child row: a foreign key constraint
fails (`netdot`.`asset`, CONSTRAINT `fk_physaddr_1` FOREIGN KEY (`physaddr`)
REFERENCES `physaddr` (`id`)) [for Statement "INSERT INTO asset (product_id,
reserved_for, physaddr)
VALUES (?, ?, ?)
" with ParamValues: 0='1', 1=undef, 2=2] at
/usr/share/perl5/DBIx/ContextualFetch.pm line 52.
at /usr/local/netdot/bin/updatedevices.pl line 213.
</pre>
So two issues.
Problem 1: error handling when no base MAC address can be found (shouldn't try
to insert undef value; shouldn't continue with a SQL constraint failure). I
haven't looked at ways of improving this. Note that recent fix #1814 will
suppress the sprintf error, but I'm not sure it will fix the Asset insertion.
Problem 2: why no base MAC address was found on a machine which clearly has
interfaces?
<pre>
$ snmpwalk -v2c -c XXXX x.x.x.21 ifTable
IF-MIB::ifIndex.1 = INTEGER: 1
IF-MIB::ifIndex.2 = INTEGER: 2
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: em1
IF-MIB::ifType.1 = INTEGER: softwareLoopback(24)
IF-MIB::ifType.2 = INTEGER: ethernetCsmacd(6)
IF-MIB::ifMtu.1 = INTEGER: 65536
IF-MIB::ifMtu.2 = INTEGER: 1500
IF-MIB::ifSpeed.1 = Gauge32: 10000000
IF-MIB::ifSpeed.2 = Gauge32: 100000000
IF-MIB::ifPhysAddress.1 = STRING:
IF-MIB::ifPhysAddress.2 = STRING: 90:b1:1c:90:d2:1c
</pre>
Note that because this machine has a modern 3.13 kernel, it is using the new
"predictable network interface names" instead of eth0. See
http://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
I will skip all the boring debugging I had to do to get to the bottom of this,
which in fact is explained in a lowly debug message:
<pre>
DEBUG - Device::get_snmp_info: x.x.x.21: Interface em1 ignored per config
option (IFRESERVED)
</pre>
Grr. The pattern explicitly excludes <code>em\d+</code>. But why? Also, the
default IFRESERVED regexp is missing parentheses, so only the first item is
anchored.
So I would suggest a better default to be:
<pre>
IFRESERVED =>
'^(bme|pime|mtun|lsi|jsrv|pimd|vt-|tap|pe-|pd-|dsc|rptr|unrouted)',
</pre>
However this still assumes that we won't ever come across any Linux devices
with names like bme0 or dsc0.
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://osl.uoregon.edu/redmine/my/account
------------------------------
Message: 2
Date: Fri, 27 Jun 2014 10:34:09 -0700
From: [email protected]
Subject: [Netdot-devel] [Netdot - Feature #1824] (Closed) Improve
Depth Up / Depth Down display
To: [email protected], [email protected]
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
Issue #1824 has been updated by Carlos Vicente.
Status changed from In Progress to Closed
Brian Candler wrote:
> Oops, I forgot to swap the head/tail labels over for the "up" direction. I
> can see this now that the lines can be made longer :-)
>
> Fix: https://github.com/cvicente/Netdot/pull/21
Merged. Thanks.
----------------------------------------
Feature #1824: Improve Depth Up / Depth Down display
https://osl.uoregon.edu/redmine/issues/1824#change-3208
Author: Brian Candler
Status: Closed
Priority: Normal
Assignee: Brian Candler
Category: UserInterface
Target version: 1.0.7
Resolution: fixed
When viewing the topology from the view of at a particular device, you can
select "Depth Up" (towards the NMS_DEVICE) and "Depth Down" (away from the
NMS_DEVICE)
This works, but it displays a tree rooted at the chosen device on the left and
both "Up" and "Down" devices mixed together to the right.
What I was expecting was a chain of "Up" devices to be on the left, and the
"Down" devices to the right.
In the attached example diagram, "lib-core" is actually the parent (closer to
NMS_DEVICE) and the other devices are children, but you can't tell this from
the diagram.
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://osl.uoregon.edu/redmine/my/account
------------------------------
Message: 3
Date: Fri, 27 Jun 2014 10:57:34 -0700
From: [email protected]
Subject: [Netdot-devel] [SCM] Netdot branch master updated.
netdot-1.0.6-24-g2c3925c
To: [email protected]
Message-ID: <[email protected]>
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "Netdot".
The branch, master has been updated
via 2c3925ce8fce1880c2a718e661380665a5332484 (commit)
from 22ddc22ecf208ee5fdcc93f249e53ab362b71334 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
commit 2c3925ce8fce1880c2a718e661380665a5332484
Author: Carlos Vicente <[email protected]>
Date: Fri Jun 27 13:57:09 2014 -0400
Fix for #1791 (Exclude static IP information from prune_db.pl's IP address
pruning)
diff --git a/bin/prune_db.pl b/bin/prune_db.pl
index c852006..5a52b29 100755
--- a/bin/prune_db.pl
+++ b/bin/prune_db.pl
@@ -1,9 +1,8 @@
#!<<Make:PERL>>
-###############################################################
-# prune_db.pl
+# Prune database
+#
#
-
use lib "<<Make:LIB>>";
use Netdot::Model;
use Netdot::Config;
@@ -42,10 +41,10 @@ my $usage = <<EOF;
-h, --help Print help
EOF
-
+
# handle cmdline args
-my $result = GetOptions(
+my $result = GetOptions(
"F|fwt" => \$self{FWT},
"A|arp" => \$self{ARP},
"M|macs" => \$self{MACS},
@@ -96,7 +95,7 @@ $logger->debug(sprintf("NUM_DAYS(%d) ago was : %s",
$self{NUM_DAYS}, $sqldate));
my $start = time;
my %rows_deleted;
-
+
if ( $self{MACS} ){
###########################################################################################
# Delete MAC addresses that don't belong to devices (static flag is off)
@@ -112,12 +111,11 @@ if ( $self{MACS} ){
}
if ( $self{IPS} ){
###########################################################################################
- # Delete 'Discovered' and 'Static' IP addresses
+ # Delete 'Discovered' IP addresses
# Note: This will also delete A/AAAA records, ArpCache entries,
DhcpScopes, etc.
my $q = $dbh->prepare("SELECT ipblock.id
FROM ipblock, ipblockstatus
- WHERE (ipblockstatus.name='Discovered'
- OR ipblockstatus.name='Static')
+ WHERE ipblockstatus.name='Discovered'
AND ipblock.status=ipblockstatus.id
AND ipblock.last_seen < ?");
$q->execute($sqldate);
@@ -140,7 +138,7 @@ if ( $self{RR} ){
expiration => {'<>', '1970-01-01'},
expiration => {'<>', '1970-01-02'}]
);
-
+
my @rrs = RR->search_where(@where);
unless ( $self{PRETEND} ){
@@ -149,7 +147,7 @@ if ( $self{RR} ){
$rr->delete();
}
}
-
+
$rows_deleted{rr} = scalar(@rrs);
}
@@ -207,7 +205,7 @@ if ( $self{ARP} ){
# Delete ArpCaches
# Note: This will also delete ArpCacheEntry objects.
$logger->info("Deleting ARP Caches older than $sqldate");
-
+
my @arps = ArpCache->search_where(tstamp=>{ '<', $sqldate });
foreach my $arp ( @arps ){
$logger->debug("Deleting ArpCache id ". $arp->id);
@@ -244,13 +242,13 @@ sub optimize_table{
my ($table) = @_;
if ( $db_type eq 'mysql' ){
- $dbh->do("OPTIMIZE TABLE $table");
+ $dbh->do("OPTIMIZE TABLE $table");
}elsif( $db_type eq 'Pg' ){
- $dbh->do("VACUUM $table");
+ $dbh->do("VACUUM $table");
}else{
$logger->warn("Could not optimize table $table. Database $db_type not
supported");
- }
-
+ }
+
return;
}
@@ -275,13 +273,13 @@ sub rotate_tables{
my $db_user = Netdot->config->get('DB_DBA');
my $db_pass = Netdot->config->get('DB_DBA_PASSWORD');
my $db_db = Netdot->config->get('DB_DATABASE');
-
+
my $dbh = &dbconnect($db_type, $db_host, $db_port, $db_user, $db_pass,
$db_db)
|| die ("Cannot connect to database as root");
-
+
$dbh->{AutoCommit} = 0; # make sure autocommit is off so we use
transactions
$dbh->{RaiseError} = 1; # make sure we hear about any problems
-
+
my $timestamp = time;
my %defs;
@@ -311,7 +309,7 @@ sub rotate_tables{
$logger->info("Tables $table_list rotated successfully");
# We can turn autocommit back on since the rest of the transactions are
basically atomic
$dbh->{AutoCommit} = 1;
-
+
&dbdisconnect($dbh);
return 1;
}
-----------------------------------------------------------------------
Summary of changes:
bin/prune_db.pl | 38 ++++++++++++++++++--------------------
1 file changed, 18 insertions(+), 20 deletions(-)
hooks/post-receive
--
Netdot
------------------------------
Message: 4
Date: Fri, 27 Jun 2014 10:57:56 -0700
From: [email protected]
Subject: [Netdot-devel] [SCM] Netdot branch netdot-1.0 updated.
netdot-1.0.6-24-g2c3925c
To: [email protected]
Message-ID: <[email protected]>
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "Netdot".
The branch, netdot-1.0 has been updated
via 2c3925ce8fce1880c2a718e661380665a5332484 (commit)
from 22ddc22ecf208ee5fdcc93f249e53ab362b71334 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
-----------------------------------------------------------------------
Summary of changes:
bin/prune_db.pl | 38 ++++++++++++++++++--------------------
1 file changed, 18 insertions(+), 20 deletions(-)
hooks/post-receive
--
Netdot
------------------------------
Message: 5
Date: Fri, 27 Jun 2014 10:59:06 -0700
From: [email protected]
Subject: [Netdot-devel] [Netdot - Feature #1791] (Resolved) Exclude
static IP information from prune_db.pl's IP address pruning
To: [email protected], [email protected]
Cc: [email protected]
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
Issue #1791 has been updated by Carlos Vicente.
Status changed from New to Resolved
Assignee set to Carlos Vicente
Target version set to 1.0.7
Resolution set to fixed
The recommendation seems appropriate. Changes committed towards next release.
Thanks.
----------------------------------------
Feature #1791: Exclude static IP information from prune_db.pl's IP address
pruning
https://osl.uoregon.edu/redmine/issues/1791#change-3209
Author: Timothy Snowberger
Status: Resolved
Priority: Normal
Assignee: Carlos Vicente
Category: AddressTracking
Target version: 1.0.7
Resolution: fixed
Currently, prune_db.pl appears to include 'Discovered' and 'Static' IPs as part
of the normal IP pruning process.
Scenario:
Assume you have a routing device which does not report back ARP information,
such as a non-Cisco device (or outdated Cisco device). A user then creates
static IP assignments within a block inside of netdot manually by hand, which
are not associated with actively polled devices, just description information.
If you run prune_db.pl -M -I -d 90, these items are eventually deleted because
they have a last seen time of whenever the static IP information was entered.
We have a couple of subnets behind old routing devices that do not report ARP
tables via SNMP, and would require some work to gather ARP information via CLI.
As I'd still like to be able to prune discovered IPs, I'd like to suggest that
either an option be added to prune_db.pl to exclude static IPs from the normal
pruning process, or make an option within the netdot GUI that prevents deletion
of static IPs that should never be removed, whether they are seen or not. I
realize that marking IPs as "Reserved" could be another solution, but I prefer
to label them as "Static".
This appears to be the relevant section of code in prune_db.pl (lines 167 - 187:
<pre>
if ( $self{IPS} ){
###########################################################################################
# Delete 'Discovered' and 'Static' IP addresses
# Note: This will also delete A/AAAA records, ArpCache entries, DhcpScopes,
etc.
my $q = $dbh->prepare("SELECT ipblock.id
FROM ipblock, ipblockstatus
WHERE (ipblockstatus.name='Discovered'
OR ipblockstatus.name='Static')
AND ipblock.status=ipblockstatus.id
AND ipblock.last_seen < ?");
$q->execute($sqldate);
while ( my $id = $q->fetchrow_array() ) {
if ( my $ip = Ipblock->retrieve($id) ){
$logger->debug(sprintf("Deleting IP %s", $ip->address));
unless ( $self{PRETEND} ){
$ip->delete() ;
$rows_deleted{ipblock}++;
}
}
}
}
</pre>
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://osl.uoregon.edu/redmine/my/account
------------------------------
Message: 6
Date: Fri, 27 Jun 2014 11:59:43 -0700
From: [email protected]
Subject: [Netdot-devel] [Netdot - Bug #1818] Could not insert base
MAC: : Error while inserting PhysAddr: Missing address
To: [email protected], [email protected]
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
Issue #1818 has been updated by Brian Candler.
Carlos Vicente wrote:
> Problem 1:
>
> Very strange because notice that the physaddr value passed in the SQL insert
> statement is "2". I re-read the code and could not figure out how that may
> have happened.
I did have a capture of $info in _assign_base_mac but unfortunately haven't
kept the logs.
However IIRC, the interfaces at that point looked something like this:
<pre>
1 => { full set of interface info including name "lo", MAC and IPv6 addrs,
although ISTR IPv4 address 127.0.0.1 was missing }
2 => { just the IPv6 link-local address, no other info }
</pre>
----------------------------------------
Bug #1818: Could not insert base MAC: : Error while inserting PhysAddr: Missing
address
https://osl.uoregon.edu/redmine/issues/1818#change-3210
Author: Brian Candler
Status: Resolved
Priority: Normal
Assignee: Brian Candler
Category: DeviceManagement
Target version: 1.0.7
Resolution: fixed
[Netdot 1.0.6, configured with IFNAME_SHORT => 1, running under Ubuntu 14.04]
This is when adding the first device, which is the Netdot server itself,
running snmpd.
It was added via the GUI and using SNMP, and then I ran from the CLI:
<pre>
/usr/local/netdot/bin/updatedevices.pl -DIFAT -d
...
DEBUG - Device::get_snmp_info: Finished getting SNMP info from x.x.x.21
DEBUG - monitor.example.net did not return base MAC. Using first available
interface MAC.
Use of uninitialized value $address in sprintf at
/usr/local/netdot/lib/Netdot/Model/Device.pm line 5641.
DEBUG - monitor.example.net: Could not insert base MAC: : Error while inserting
PhysAddr: Missing address
DEBUG - monitor.example.net: No suitable base MAC found
ERROR - Error while inserting Asset: Can't insert new Asset: DBD::mysql::st
execute failed: Cannot add or update a child row: a foreign key constraint
fails (`netdot`.`asset`, CONSTRAINT `fk_physaddr_1` FOREIGN KEY (`physaddr`)
REFERENCES `physaddr` (`id`)) [for Statement "INSERT INTO asset (product_id,
reserved_for, physaddr)
VALUES (?, ?, ?)
" with ParamValues: 0='1', 1=undef, 2=2] at
/usr/share/perl5/DBIx/ContextualFetch.pm line 52.
at /usr/local/netdot/bin/updatedevices.pl line 213.
</pre>
So two issues.
Problem 1: error handling when no base MAC address can be found (shouldn't try
to insert undef value; shouldn't continue with a SQL constraint failure). I
haven't looked at ways of improving this. Note that recent fix #1814 will
suppress the sprintf error, but I'm not sure it will fix the Asset insertion.
Problem 2: why no base MAC address was found on a machine which clearly has
interfaces?
<pre>
$ snmpwalk -v2c -c XXXX x.x.x.21 ifTable
IF-MIB::ifIndex.1 = INTEGER: 1
IF-MIB::ifIndex.2 = INTEGER: 2
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: em1
IF-MIB::ifType.1 = INTEGER: softwareLoopback(24)
IF-MIB::ifType.2 = INTEGER: ethernetCsmacd(6)
IF-MIB::ifMtu.1 = INTEGER: 65536
IF-MIB::ifMtu.2 = INTEGER: 1500
IF-MIB::ifSpeed.1 = Gauge32: 10000000
IF-MIB::ifSpeed.2 = Gauge32: 100000000
IF-MIB::ifPhysAddress.1 = STRING:
IF-MIB::ifPhysAddress.2 = STRING: 90:b1:1c:90:d2:1c
</pre>
Note that because this machine has a modern 3.13 kernel, it is using the new
"predictable network interface names" instead of eth0. See
http://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
I will skip all the boring debugging I had to do to get to the bottom of this,
which in fact is explained in a lowly debug message:
<pre>
DEBUG - Device::get_snmp_info: x.x.x.21: Interface em1 ignored per config
option (IFRESERVED)
</pre>
Grr. The pattern explicitly excludes <code>em\d+</code>. But why? Also, the
default IFRESERVED regexp is missing parentheses, so only the first item is
anchored.
So I would suggest a better default to be:
<pre>
IFRESERVED =>
'^(bme|pime|mtun|lsi|jsrv|pimd|vt-|tap|pe-|pd-|dsc|rptr|unrouted)',
</pre>
However this still assumes that we won't ever come across any Linux devices
with names like bme0 or dsc0.
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://osl.uoregon.edu/redmine/my/account
------------------------------
_______________________________________________
Netdot-devel mailing list
[email protected]
https://osl.uoregon.edu/mailman/listinfo/netdot-devel
End of Netdot-devel Digest, Vol 87, Issue 13
********************************************
