I've updated the server to amanda-backup_server-3.5-1 (64bit) which appears to have fixed the issue. The client that failed most regularly is running amanda-backup_client-3.3.9-1 (32bit).
I'll keep monitoring this in case the situation changes but it looks like it's working properly now. On 05/10/17 08:58, Tom Robinson wrote: > > It may well be just that I can't see the wood for the trees when looking at > logging but I can't > find the problem :-( > > I'm running daily manual dumps of the FAILED DLE's to keep backups intact! > > I'm still getting the following: > > FAILURE DUMP SUMMARY: > bentley Resources lev 1 FAILED [too many dumper retry: [request failed: No > route to host]] > bentley sysadmin lev 1 FAILED [too many dumper retry: [request failed: No > route to host]] > > Apart from the two KVM hosts, all these systems are KVM Guests. The backup > server is a KVM guest. > Has anyone seen or know of issues that may occur with amanda on virtualised > infrastructure? > > From my understanding of KVM networking between guests, whole network frames > are dumped and picked > up between them. This allows higher transport speeds. I've tested the > throughput with iperf and > have seen througput as high as 25Gbps. The following ipef session shows the > connection between the > failed guest, bentley, and the backup server. I've only shown the 'server' > side results for iperf > below: > > # systemctl stop xinetd > > # iperf -p 10080 -s > ------------------------------------------------------------ > Server listening on TCP port 10080 > TCP window size: 85.3 KByte (default) > ------------------------------------------------------------ > [ 4] local 10.0.19.21 port 10080 connected with 192.168.0.3 port 39214 > [ ID] Interval Transfer Bandwidth > [ 4] 0.0-10.0 sec 20.5 GBytes 17.6 Gbits/sec > [ 4] local 10.0.19.21 port 10080 connected with 192.168.0.3 port 39215 > [ 4] 0.0-10.0 sec 20.7 GBytes 17.8 Gbits/sec > [ 4] local 10.0.19.21 port 10080 connected with 192.168.0.3 port 39218 > [ 4] 0.0-10.0 sec 21.3 GBytes 18.3 Gbits/sec > [ 4] local 10.0.19.21 port 10080 connected with 192.168.0.3 port 39223 > [ 4] 0.0-10.0 sec 21.4 GBytes 18.4 Gbits/sec > > Any clues/help for the above are appreciated. > > I'm now also getting some other strange errors that I've never seen before. > These report as > 'FAILED' but further on into the report they appear to have completed without > issue. What do the > error codes signify (e.g. FAILED [02-00098] etc.)? > > ---8<--- > > FAILURE DUMP SUMMARY: > ---8<--- > bentley ECN lev 0 FAILED [02-00098] > bentley Repair lev 1 FAILED [06-00229] > garage /var lev 1 FAILED [shm_ring cancelled] > modena /usr/src lev 1 FAILED [12-00205] > > ---8<--- > NOTES: > planner: Last full dump of bentley:ECN on tape daily02 overwritten in 5 > runs. > planner: Last level 1 dump of bentley:ECN on tape daily01 overwritten in 4 > runs. > planner: Last full dump of bentley:Repair on tape daily07 overwritten in 2 > runs. > planner: Last full dump of garage:/var on tape daily01 overwritten in 4 > runs. > > ---8<--- > DUMP SUMMARY: > DUMPER STATS > TAPER STATS > HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS > KB/s MMM:SS KB/s > ------------------------------------ --------------------------- > --------------- --------------- > ---8<--- > bentley ECN 0 19790 19790 -- 0:03 > 7325.0 0:00 197900.0 > bentley Repair 1 10 0.0 0:00 > 4.2 0:00 0.0 > garage /var 1 7000 7000 -- 0:00 > 33341.0 0:00 70000.0 > modena /usr/src 1 190 14 7.4 0:04 > 3.3 0:00 140.0 > ---8<--- > > > What are the error codes and did amanda dump these OK or not? > > Kind regards, > Tom > > > Tom Robinson > IT Manager/System Administrator > > MoTeC Pty Ltd > > 121 Merrindale Drive > Croydon South > 3136 Victoria > Australia > > T: +61 3 9761 5050 > F: +61 3 9761 5051 > E: tom.robin...@motec.com.au > On 13/09/17 23:09, Jean-Louis Martineau wrote: >> Tom, >> >> It is the system that return the "No route to host" error. >> You should check your system log (on server, client, router, firewall, nat, >> ...) for network error. >> >> Jean-Louis >> >> On 12/09/17 06:01 PM, Tom Robinson wrote: >>> bump >>> >>> On 11/09/17 12:45, Tom Robinson wrote: >>> > Hi, >>> > >>> > I've recently migrated our backup server from CentOS 5 to CentOS 7. I've >>> > also upgraded from amanda >>> > 3.3.7 to 3.4.5 >>> > >>> > The amcheck works fine and reports no issues. Yet, on backup runs on some >>> > DLEs I get the error: >>> > >>> > dump failed: [request failed: No route to host](too) >>> > >>> > It also appears to be random as to which DLEs fail. Sometimes it's just >>> > one or two on a client. >>> > Other times it's all DLEs for a client. And, for any particular client it >>> > can be a different >>> DLE on >>> > that client each day. >>> > >>> > Below is a dumper..debug log from the server. I'm not sure what to check >>> > for in there. What other >>> > logs should I check? >>> > >>> > Kind regards, >>> > Tom >>> > >>> > Sun Sep 10 20:16:32.115899592 2017: pid 6088: thd-0x257f400: dumper: >>> > close_producer_shm_ring >>> > sem_close(sem_write 0x7fbc1588b000 >>> > Sun Sep 10 20:16:32.115911222 2017: pid 6088: thd-0x257f400: dumper: >>> > am_sem_close 0x7fbc1588b000 0 >>> > Sun Sep 10 20:16:32.115927349 2017: pid 6088: thd-0x257f400: dumper: >>> > am_sem_close 0x7fbc15889000 0 >>> > Sun Sep 10 20:16:32.115938800 2017: pid 6088: thd-0x257f400: dumper: >>> > am_sem_close 0x7fbc1588a000 0 >>> > Sun Sep 10 20:16:32.115949293 2017: pid 6088: thd-0x257f400: dumper: >>> > am_sem_close 0x7fbc15888000 0 >>> > Sun Sep 10 20:16:32.337361676 2017: pid 6088: thd-0x257f400: dumper: >>> > getcmd: SHM-DUMP 00-00217 >>> 34076 >>> > NULL 5 bentley ffffffff9efefbffffffffff3f000000000000 Dispatch >>> /var/lib/samba/data/public/Dispatch 1 >>> > 2017:9:6:4:6:22 GNUTAR "" "" "" "" "" "" "" 1 "" "" bsdtcp AMANDA /amand >>> > a_shm_control-6956-0 20 |" <auth>bsdtcp</auth>\n >>> > <compress>FAST</compress>\n >>> > <record>YES</record>\n <index>YES</index>\n >>> > <datapath>AMANDA</datapath>\n""" >>> > Sun Sep 10 20:16:32.337507787 2017: pid 6088: thd-0x257f400: dumper: >>> > Sending header to >>> localhost:34076 >>> > Sun Sep 10 20:16:32.339939372 2017: pid 6088: thd-0x257f400: dumper: >>> > make_socket opening >>> socket with >>> > family 10 >>> > Sun Sep 10 20:16:32.339978452 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_port: Try port 1024: >>> > available - Success >>> > Sun Sep 10 20:16:32.340075462 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_portrange: >>> Connect from >>> > :::1024 failed: Connection refused >>> > Sun Sep 10 20:16:32.340101209 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_portrange: connect to >>> > ::1:34076 failed: Connection refused >>> > Sun Sep 10 20:16:32.342383119 2017: pid 6088: thd-0x257f400: dumper: >>> > make_socket opening >>> socket with >>> > family 2 >>> > Sun Sep 10 20:16:32.342418634 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_port: Try port 1024: >>> > available - Success >>> > Sun Sep 10 20:16:32.342489613 2017: pid 6088: thd-0x257f400: dumper: >>> > connected to >>> 127.0.0.1:34076 <http://127.0.0.1:34076> >>> > Sun Sep 10 20:16:32.342501059 2017: pid 6088: thd-0x257f400: dumper: our >>> > side is 0.0.0.0:1024 >>> <http://0.0.0.0:1024> >>> > Sun Sep 10 20:16:32.342509347 2017: pid 6088: thd-0x257f400: dumper: >>> > try_socksize: send buffer >>> size >>> > is 131072 >>> > Sun Sep 10 20:16:32.342558663 2017: pid 6088: thd-0x257f400: dumper: send >>> > request: >>> > ---- >>> > SERVICE sendbackup >>> > OPTIONS >>> > features=ffffffff9efefbfffffffffffffff3fffbf70f;maxdumps=5;hostname=bentley;config=daily; >>> > <dle> >>> > <program>GNUTAR</program> >>> > <disk>Dispatch</disk> >>> > <diskdevice>/var/lib/samba/data/public/Dispatch</diskdevice> >>> > <level>1</level> >>> > <auth>bsdtcp</auth> >>> > <compress>FAST</compress> >>> > <record>YES</record> >>> > <index>YES</index> >>> > <datapath>AMANDA</datapath> >>> > </dle> >>> > >>> > ---- >>> > >>> > Sun Sep 10 20:16:32.342572947 2017: pid 6088: thd-0x257f400: dumper: >>> security_getdriver(name=bsdtcp) >>> > returns 0x7fbc153e86a0 >>> > Sun Sep 10 20:16:32.342582472 2017: pid 6088: thd-0x257f400: dumper: >>> > security_handleinit(handle=0x25e2e70, driver=0x7fbc153e86a0 (BSDTCP)) >>> > Sun Sep 10 20:16:32.343623490 2017: pid 6088: thd-0x257f400: dumper: >>> > security_streaminit(stream=0x283d6e0, driver=0x7fbc153e86a0 (BSDTCP)) >>> > Sun Sep 10 20:16:32.346176806 2017: pid 6088: thd-0x257f400: dumper: >>> > make_socket opening >>> socket with >>> > family 2 >>> > Sun Sep 10 20:16:32.346230063 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_port: Try port 571: >>> > available - Success >>> > Sun Sep 10 20:16:32.346247716 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_portrange: >>> Connect from >>> > 0.0.0.0:571 <http://0.0.0.0:571> failed: Cannot assign requested address >>> > Sun Sep 10 20:16:32.346261235 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_portrange: connect to >>> > 192.168.0.3:10080 <http://192.168.0.3:10080> failed: Cannot assign >>> > requested address >>> > Sun Sep 10 20:16:32.348492651 2017: pid 6088: thd-0x257f400: dumper: >>> > make_socket opening >>> socket with >>> > family 2 >>> > Sun Sep 10 20:16:32.348526207 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_port: Try port 585: >>> > available - Success >>> > Sun Sep 10 20:18:39.587177652 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_portrange: >>> Connect from >>> > 0.0.0.0:585 <http://0.0.0.0:585> failed: Connection timed out >>> > Sun Sep 10 20:18:39.587235409 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_portrange: connect to >>> > 192.168.0.3:10080 <http://192.168.0.3:10080> failed: Connection timed out >>> > Sun Sep 10 20:18:39.587267623 2017: pid 6088: thd-0x257f400: dumper: >>> > stream_client: Could not bind >>> > to port in range 512-1023. >>> > Sun Sep 10 20:18:39.587290672 2017: pid 6088: thd-0x257f400: dumper: >>> > security_seterror(handle=0x25e2e70, driver=0x7fbc153e86a0 (BSDTCP) >>> > error=Connection timed out) >>> > Sun Sep 10 20:18:39.587299769 2017: pid 6088: thd-0x257f400: dumper: >>> > security_close(handle=0x25e2e70, driver=0x7fbc153e86a0 (BSDTCP)) >>> > Sun Sep 10 20:18:39.587308804 2017: pid 6088: thd-0x257f400: dumper: >>> security_stream_close(0x283d6e0) >>> > Sun Sep 10 20:18:44.592385871 2017: pid 6088: thd-0x257f400: dumper: >>> > security_handleinit(handle=0x25def50, driver=0x7fbc153e86a0 (BSDTCP)) >>> > Sun Sep 10 20:18:44.593300425 2017: pid 6088: thd-0x257f400: dumper: >>> > security_streaminit(stream=0x2845d10, driver=0x7fbc153e86a0 (BSDTCP)) >>> > Sun Sep 10 20:18:44.596042201 2017: pid 6088: thd-0x257f400: dumper: >>> > make_socket opening >>> socket with >>> > family 2 >>> > Sun Sep 10 20:18:44.596123530 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_port: Try port 571: >>> > available - Success >>> > Sun Sep 10 20:18:44.596155627 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_portrange: >>> Connect from >>> > 0.0.0.0:571 <http://0.0.0.0:571> failed: Cannot assign requested address >>> > Sun Sep 10 20:18:44.596163545 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_portrange: connect to >>> > 192.168.0.3:10080 <http://192.168.0.3:10080> failed: Cannot assign >>> > requested address >>> > Sun Sep 10 20:18:44.598648813 2017: pid 6088: thd-0x257f400: dumper: >>> > make_socket opening >>> socket with >>> > family 2 >>> > Sun Sep 10 20:18:44.598686408 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_port: Try port 585: >>> > available - Success >>> > Sun Sep 10 20:18:44.598811157 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_portrange: >>> Connect from >>> > 0.0.0.0:585 <http://0.0.0.0:585> failed: No route to host >>> > Sun Sep 10 20:18:44.598823545 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_portrange: connect to >>> > 192.168.0.3:10080 <http://192.168.0.3:10080> failed: No route to host >>> > Sun Sep 10 20:18:44.598852583 2017: pid 6088: thd-0x257f400: dumper: >>> > stream_client: Could not bind >>> > to port in range 512-1023. >>> > Sun Sep 10 20:18:44.598866659 2017: pid 6088: thd-0x257f400: dumper: >>> > security_seterror(handle=0x25def50, driver=0x7fbc153e86a0 (BSDTCP) >>> > error=No route to host) >>> > Sun Sep 10 20:18:44.598873575 2017: pid 6088: thd-0x257f400: dumper: >>> > security_close(handle=0x25def50, driver=0x7fbc153e86a0 (BSDTCP)) >>> > Sun Sep 10 20:18:44.598880166 2017: pid 6088: thd-0x257f400: dumper: >>> security_stream_close(0x2845d10) >>> > Sun Sep 10 20:18:49.603142410 2017: pid 6088: thd-0x257f400: dumper: >>> > security_handleinit(handle=0x25def50, driver=0x7fbc153e86a0 (BSDTCP)) >>> > Sun Sep 10 20:18:49.604135032 2017: pid 6088: thd-0x257f400: dumper: >>> > security_streaminit(stream=0x284e340, driver=0x7fbc153e86a0 (BSDTCP)) >>> > Sun Sep 10 20:18:49.606574107 2017: pid 6088: thd-0x257f400: dumper: >>> > make_socket opening >>> socket with >>> > family 2 >>> > Sun Sep 10 20:18:49.606626709 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_port: Try port 571: >>> > available - Success >>> > Sun Sep 10 20:18:49.606645124 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_portrange: >>> Connect from >>> > 0.0.0.0:571 <http://0.0.0.0:571> failed: Cannot assign requested address >>> > Sun Sep 10 20:18:49.606652941 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_portrange: connect to >>> > 192.168.0.3:10080 <http://192.168.0.3:10080> failed: Cannot assign >>> > requested address >>> > Sun Sep 10 20:18:49.609070482 2017: pid 6088: thd-0x257f400: dumper: >>> > make_socket opening >>> socket with >>> > family 2 >>> > Sun Sep 10 20:18:49.609113309 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_port: Try port 585: >>> > available - Success >>> > Sun Sep 10 20:18:49.609232279 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_portrange: >>> Connect from >>> > 0.0.0.0:585 <http://0.0.0.0:585> failed: No route to host >>> > Sun Sep 10 20:18:49.609244774 2017: pid 6088: thd-0x257f400: dumper: >>> > connect_portrange: connect to >>> > 192.168.0.3:10080 <http://192.168.0.3:10080> failed: No route to host >>> > Sun Sep 10 20:18:49.609256366 2017: pid 6088: thd-0x257f400: dumper: >>> > stream_client: Could not bind >>> > to port in range 512-1023. >>> > Sun Sep 10 20:18:49.609267524 2017: pid 6088: thd-0x257f400: dumper: >>> > security_seterror(handle=0x25def50, driver=0x7fbc153e86a0 (BSDTCP) >>> > error=No route to host) >>> > Sun Sep 10 20:18:49.609276028 2017: pid 6088: thd-0x257f400: dumper: >>> > security_close(handle=0x25def50, driver=0x7fbc153e86a0 (BSDTCP)) >>> > Sun Sep 10 20:18:49.609282311 2017: pid 6088: thd-0x257f400: dumper: >>> security_stream_close(0x284e340) >>> > Sun Sep 10 20:18:49.609296611 2017: pid 6088: thd-0x257f400: dumper: >>> > putresult: 11 TRY-AGAIN >>> > 00-00217 "[request failed: No route to host]" >>> > >>> >>> >> >> >> >> *Disclaimer* >> >> This message is the property of *CARBONITE, INC.* <http://www.carbonite.com> >> and may contain >> confidential or privileged information. >> >> If this message has been delivered to you by mistake, then do not copy or >> deliver this message to >> anyone. Instead, destroy it and notify me by reply e-mail. >> >
signature.asc
Description: OpenPGP digital signature
