Send netdisco-users mailing list submissions to
netdisco-users@lists.sourceforge.net
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.sourceforge.net/lists/listinfo/netdisco-users
or, via email, send a message with subject or body 'help' to
netdisco-users-requ...@lists.sourceforge.net
You can reach the person managing the list at
netdisco-users-ow...@lists.sourceforge.net
When replying, please edit your Subject line so it is more specific
than "Re: Contents of netdisco-users digest..."
Today's Topics:
1. Out of memory errors in backend log upon upgrading to
Netdisco 2.60.7 (Muris)
2. Re: Out of memory errors in backend log upon upgrading to
Netdisco 2.60.7 (Oliver Gorwits)
--- Begin Message ---
Hey guys,
I seem to be having some out of memory errors on upgrading to netdisco 2.60.7
the latest version.. not sure whats going on.
I followed this guide
https://severalnines.com/blog/architecture-and-tuning-memory-postgresql-databases/
And I increased the worker_mem to 32MB which seems to have helped a bit, but
still noticing errors on the backend time to time, and messages like this, any
ideas?
Running on Xeon 8 core with 16gb ram.
DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute
failed: ERROR: out of memory
DETAIL: Failed on request of size 32800 in memory context "HashBatchContext".
[for Statement "SELECT me.job, me.entered, me.started, me.finished, me.device,
me.port, me.action, me.subaction, me.status, me.username, me.userip, me.log,
me.debug, me.device_key, me.job_priority FROM ( WITH my_jobs AS
(SELECT admin.* FROM admin
LEFT OUTER JOIN device_skip ds
ON (ds.backend = ? AND admin.device = ds.device
AND admin.action = ANY (ds.actionset))
WHERE admin.status = 'queued'
AND ds.device IS NULL)
SELECT my_jobs.*,
CASE WHEN ( (my_jobs.username IS NOT NULL AND ((ds.deferrals IS NULL
AND ds.last_defer IS NULL)
OR my_jobs.entered >
ds.last_defer))
OR (my_jobs.action = ANY (string_to_array(btrim(?, '{"}'),
'","'))) )
THEN 100
ELSE 0
END AS job_priority
FROM my_jobs
LEFT OUTER JOIN device_skip ds
ON (ds.backend = ? AND ds.device = my_jobs.device)
WHERE ds.deferrals < ?
OR (my_jobs.username IS NOT NULL AND (ds.last_defer IS NULL
OR my_jobs.entered > ds.last_defer))
OR (ds.deferrals IS NULL AND ds.last_defer IS NULL)
OR ds.last_defer <= ( LOCALTIMESTAMP - ?::interval )
ORDER BY job_priority DESC,
ds.deferrals ASC NULLS FIRST,
ds.last_defer ASC NULLS LAST,
device_key DESC NULLS LAST,
random()
LIMIT ?
) me" with ParamValues: 1='netdisco',
2='{"contact","hook::exec","hook::http","location","portcontrol","portname","power","snapshot","vlan"}',
3=’netdisco’, 4='10', 5='7 days', 6='1'] at
/home/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 122
[5405] 2023-03-07 08:36:21 error bless( {'msg' =>
'DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute
failed: ERROR: out of memory
DETAIL: Failed on request of size 72 in memory context "MessageContext". [for
Statement "UPDATE admin SET finished = ?, log = ?, started = ?, status = ?
WHERE ( job = ? )" with ParamValues: 1=\'Tue Mar 7 19:06:21 2023\',
2=\'DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st
execute failed: ERROR: out of memory
DETAIL: Failed on request of size 16 in memory context "MessageContext". [for
Statement "SELECT me.ip, me.port, me.creation, me.descr, me.up, me.up_admin,
me.type, me.duplex, me.duplex_admin, me.speed, me.speed_admin, me.name, me.mac,
me.mtu, me.stp, me.remote_ip, me.remote_port, me.remote_type, me.remote_id,
me.has_subinterfaces, me.is_master, me.slave_of, me.manual_topo, me.is_uplink,
me.vlan, me.pvid, me.lastchange, me.custom_fields, neighbor_alias.ip,
neighbor_alias.alias, neighbor_alias.subnet, neighbor_alias.port,
neighbor_alias.dns, neighbor_alias.creation, device.ip, device.creation,
device.dns, device.description, device.uptime, device.contact, device.name,
device.location, device.layers, device.num_ports, device.mac, device.serial,
device.chassis_id, device.model, device.ps1_type, device.ps2_type,
device.ps1_status, device.ps2_status, device.fan, device.slots, dev...\',
3=\'Tue Mar 7 19:06:17 2023\', 4=\'error\', 5=\'215666754\'] at
/home/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 282
'}, 'DBIx::Class::Exception' )
[5427] 2023-03-07 08:36:28 error bless( {'msg' =>
'DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI
connect(\'dbname=netdisco\',\'netdisco\',...) failed: server closed the
connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request. at
/home/netdisco/perl5/lib/perl5/DBIx/Class/Storage/DBI.pm line 1639. at
/home/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 292
'}, 'DBIx::Class::Exception' )
DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute
failed: ERROR: out of memory
Use of uninitialized value $op in pattern match (m//) at
/home/netdisco/perl5/lib/perl5/SQL/Abstract/Classic.pm line 327.
[14006] 2023-03-07 10:05:49 error bless( {'msg' =>
'DBIx::Class::SQLMaker::ClassicExtensions::puke(): Fatal: Operator calls in
update must be in the form { -op => $arg } at
/home/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 282
'}, 'DBIx::Class::Exception' )
Use of uninitialized value $op in pattern match (m//) at
/home/netdisco/perl5/lib/perl5/SQL/Abstract/Classic.pm line 327.
[14718] 2023-03-07 10:10:16 error bless( {'msg' =>
'DBIx::Class::SQLMaker::ClassicExtensions::puke(): Fatal: Operator calls in
update must be in the form { -op => $arg } at
/home/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 282
'}, 'DBIx::Class::Exception' )
--- End Message ---
--- Begin Message ---
Thanks for raising the issue, Muris. I will look into it.
Couple of questions: how many rows are in your "admin" table in the
database (this is the job queue).
Also, have you been using the snapshot or SNMP browser feature?
Oh, and which version did you upgrade _from_ ?
Many thanks,
Oliver.
On Tue, 7 Mar 2023 at 10:21, Muris <alcat...@gmail.com> wrote:
> Hey guys,
>
>
>
> I seem to be having some out of memory errors on upgrading to netdisco
> 2.60.7 the latest version.. not sure whats going on.
>
>
>
> I followed this guide
> https://severalnines.com/blog/architecture-and-tuning-memory-postgresql-databases/
>
>
>
> And I increased the worker_mem to 32MB which seems to have helped a bit,
> but still noticing errors on the backend time to time, and messages like
> this, any ideas?
>
>
>
> Running on Xeon 8 core with 16gb ram.
>
>
>
> DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st
> execute failed: ERROR: out of memory
>
> DETAIL: Failed on request of size 32800 in memory context
> "HashBatchContext". [for Statement "SELECT me.job, me.entered, me.started,
> me.finished, me.device, me.port, me.action, me.subaction, me.status,
> me.username, me.userip, me.log, me.debug, me.device_key, me.job_priority
> FROM ( WITH my_jobs AS
>
> (SELECT admin.* FROM admin
>
> LEFT OUTER JOIN device_skip ds
>
> ON (ds.backend = ? AND admin.device = ds.device
>
> AND admin.action = ANY (ds.actionset))
>
> WHERE admin.status = 'queued'
>
> AND ds.device IS NULL)
>
>
>
> SELECT my_jobs.*,
>
> CASE WHEN ( (my_jobs.username IS NOT NULL AND ((ds.deferrals IS
> NULL AND ds.last_defer IS NULL)
>
> OR
> my_jobs.entered > ds.last_defer))
>
> OR (my_jobs.action = ANY (string_to_array(btrim(?,
> '{"}'), '","'))) )
>
> THEN 100
>
> ELSE 0
>
> END AS job_priority
>
> FROM my_jobs
>
>
>
> LEFT OUTER JOIN device_skip ds
>
> ON (ds.backend = ? AND ds.device = my_jobs.device)
>
>
>
> WHERE ds.deferrals < ?
>
> OR (my_jobs.username IS NOT NULL AND (ds.last_defer IS NULL
>
> OR my_jobs.entered >
> ds.last_defer))
>
> OR (ds.deferrals IS NULL AND ds.last_defer IS NULL)
>
> OR ds.last_defer <= ( LOCALTIMESTAMP - ?::interval )
>
>
>
> ORDER BY job_priority DESC,
>
> ds.deferrals ASC NULLS FIRST,
>
> ds.last_defer ASC NULLS LAST,
>
> device_key DESC NULLS LAST,
>
> random()
>
> LIMIT ?
>
> ) me" with ParamValues: 1='netdisco',
> 2='{"contact","hook::exec","hook::http","location","portcontrol","portname","power","snapshot","vlan"}',
> 3=’netdisco’, 4='10', 5='7 days', 6='1'] at
> /home/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 122
>
> [5405] 2023-03-07 08:36:21 error bless( {'msg' =>
> 'DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st
> execute failed: ERROR: out of memory
>
> DETAIL: Failed on request of size 72 in memory context "MessageContext".
> [for Statement "UPDATE admin SET finished = ?, log = ?, started = ?, status
> = ? WHERE ( job = ? )" with ParamValues: 1=\'Tue Mar 7 19:06:21 2023\',
> 2=\'DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st
> execute failed: ERROR: out of memory
>
> DETAIL: Failed on request of size 16 in memory context "MessageContext".
> [for Statement "SELECT me.ip, me.port, me.creation, me.descr, me.up,
> me.up_admin, me.type, me.duplex, me.duplex_admin, me.speed, me.speed_admin,
> me.name, me.mac, me.mtu, me.stp, me.remote_ip, me.remote_port,
> me.remote_type, me.remote_id, me.has_subinterfaces, me.is_master,
> me.slave_of, me.manual_topo, me.is_uplink, me.vlan, me.pvid, me.lastchange,
> me.custom_fields, neighbor_alias.ip, neighbor_alias.alias,
> neighbor_alias.subnet, neighbor_alias.port, neighbor_alias.dns,
> neighbor_alias.creation, device.ip, device.creation, device.dns,
> device.description, device.uptime, device.contact, device.name,
> device.location, device.layers, device.num_ports, device.mac,
> device.serial, device.chassis_id, device.model, device.ps1_type,
> device.ps2_type, device.ps1_status, device.ps2_status, device.fan,
> device.slots, dev...\', 3=\'Tue Mar 7 19:06:17 2023\', 4=\'error\',
> 5=\'215666754\'] at
> /home/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 282
>
> '}, 'DBIx::Class::Exception' )
>
>
>
> [5427] 2023-03-07 08:36:28 error bless( {'msg' =>
> 'DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI
> connect(\'dbname=netdisco\',\'netdisco\',...) failed: server closed the
> connection unexpectedly
>
> This probably means the server terminated abnormally
>
> before or while processing the request. at
> /home/netdisco/perl5/lib/perl5/DBIx/Class/Storage/DBI.pm line 1639. at
> /home/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 292
>
> '}, 'DBIx::Class::Exception' )
>
> DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st
> execute failed: ERROR: out of memory
>
>
>
> Use of uninitialized value $op in pattern match (m//) at
> /home/netdisco/perl5/lib/perl5/SQL/Abstract/Classic.pm line 327.
>
> [14006] 2023-03-07 10:05:49 error bless( {'msg' =>
> 'DBIx::Class::SQLMaker::ClassicExtensions::puke(): Fatal: Operator calls in
> update must be in the form { -op => $arg } at
> /home/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 282
>
> '}, 'DBIx::Class::Exception' )
>
> Use of uninitialized value $op in pattern match (m//) at
> /home/netdisco/perl5/lib/perl5/SQL/Abstract/Classic.pm line 327.
>
> [14718] 2023-03-07 10:10:16 error bless( {'msg' =>
> 'DBIx::Class::SQLMaker::ClassicExtensions::puke(): Fatal: Operator calls in
> update must be in the form { -op => $arg } at
> /home/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 282
>
> '}, 'DBIx::Class::Exception' )
>
>
>
>
>
>
> _______________________________________________
> Netdisco mailing list
> netdisco-users@lists.sourceforge.net
> https://sourceforge.net/p/netdisco/mailman/netdisco-users/
>
--- End Message ---
_______________________________________________
Netdisco mailing list - Digest Mode
netdisco-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/netdisco-users
