Re: ERROR: client: [DUMP program not available]

[Heiko Schlittermann]

Hi Tom,

Tom Robinson  (Mi 23 Sep 2015 01:05:39 CEST):
> CentOS 5, amanda server 2.6.0p2-1.rhel5
> CentOS 7, amanda client 3.3.3-13.el7
> 
> I'm configuring a new client into our backups with two GNUTAR based DLEs and 
> one DUMP based DLE. The
> DUMP based one fails:
…

Once I was writing a 'dump' Amanda plugin, until I discovered that 

- support from maintainer of Dump just does not exist
- Dump seems to be a dead project

And from my point of view fatally:

- Dump created dumps that restore couldn't open. It wasn't
  restores's fault, the dumps where already output broken.

Thus I'd strongly recommend not to use dump for any sort of a backup you
may need one day.

Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann
-- 
 SCHLITTERMANN.de  internet & unix support -
 Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
 gnupg encrypted messages are welcome --- key ID: F69376CE -
 ! key id 7CBF764A and 972EAC9F are revoked since 2015-01  -


signature.asc
Description: Digital signature

Re: ERROR: client: [DUMP program not available]

[Stefan Piperov]

It's amazing that a utility like dump/restore, which has been part of UNIX 
since forever, can reach the state where it's considered a dead project 
and be

unsupported...

- Stefan.


On Thu, 24 Sep 2015, Heiko Schlittermann wrote:


Hi Tom,

Tom Robinson  (Mi 23 Sep 2015 01:05:39 CEST):

CentOS 5, amanda server 2.6.0p2-1.rhel5
CentOS 7, amanda client 3.3.3-13.el7

I'm configuring a new client into our backups with two GNUTAR based DLEs and 
one DUMP based DLE. The
DUMP based one fails:

…

Once I was writing a 'dump' Amanda plugin, until I discovered that

   - support from maintainer of Dump just does not exist
   - Dump seems to be a dead project

And from my point of view fatally:

   - Dump created dumps that restore couldn't open. It wasn't
 restores's fault, the dumps where already output broken.

Thus I'd strongly recommend not to use dump for any sort of a backup you
may need one day.

   Best regards from Dresden/Germany
   Viele Grüße aus Dresden
   Heiko Schlittermann
--
SCHLITTERMANN.de  internet & unix support -
Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
gnupg encrypted messages are welcome --- key ID: F69376CE -
! key id 7CBF764A and 972EAC9F are revoked since 2015-01  -

Re: ERROR: client: [DUMP program not available]

[Tom Robinson]

On 25/09/15 06:24, Heiko Schlittermann wrote:
> Stefan Piperov  (Do 24 Sep 2015 22:16:12 CEST):
>> It's amazing that a utility like dump/restore, which has been part of UNIX
>> since forever, can reach the state where it's considered a dead project and
>> be
>> unsupported...
>>
>> - Stefan.
> Yes.  I'm talking about http://sourceforge.net/projects/dump/, if you
> know a dump/restore tools that's more alive, I'd appreciate any hint.
>
> (And yes, I was happy with dump/restore, until restore wasn't able to
> restore a dump.)
>
> Bugs:
> http://sourceforge.net/p/dump/bugs/
>
Agreed. The dump utility has a long history. Looking at the changelog on the 
CentOS 7 rpm package
and the sourceforge bug list I can see that Red Hat have been active in 
maintenance. Not sure if
these patches are making back to the repository at sourceforge, though. Also 
not sure how many of
the bugs they have addressed.



signature.asc
Description: OpenPGP digital signature

Re: ERROR: client: [DUMP program not available]

[Jason L Tibbitts III]

> "TR" == Tom Robinson  writes:

TR> Agreed. The dump utility has a long history. Looking at the
TR> changelog on the CentOS 7 rpm package and the sourceforge bug list I
TR> can see that Red Hat have been active in maintenance.

You can see the patches which Fedora carries here:
http://pkgs.fedoraproject.org/cgit/dump.git/tree/

Since Fedora is basically the upstream for RHEL and CentOS, that should
show you what's going on.  Which, if you look at the package log, is not
much.  There's only one open bug which is for an issue that's more
annoying than anything.

 - J<

Re: ERROR: client: [DUMP program not available]

[Heiko Schlittermann]

Stefan Piperov  (Do 24 Sep 2015 22:16:12 CEST):
> 
> It's amazing that a utility like dump/restore, which has been part of UNIX
> since forever, can reach the state where it's considered a dead project and
> be
> unsupported...
> 
> - Stefan.

Yes.  I'm talking about http://sourceforge.net/projects/dump/, if you
know a dump/restore tools that's more alive, I'd appreciate any hint.

(And yes, I was happy with dump/restore, until restore wasn't able to
restore a dump.)

Bugs:
http://sourceforge.net/p/dump/bugs/

Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann
-- 
 SCHLITTERMANN.de  internet & unix support -
 Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
 gnupg encrypted messages are welcome --- key ID: F69376CE -
 ! key id 7CBF764A and 972EAC9F are revoked since 2015-01  -


signature.asc
Description: Digital signature

Re: ERROR: client: [DUMP program not available]

[Tom Robinson]

On 25/09/15 08:52, Tom Robinson wrote:
> On 24/09/15 08:23, Tom Robinson wrote:
>> Hi Paul,
>>
>> Thanks and, yes, that was it. The community package to which you refer works.
> Well, I spoke too soon. I neglected to remember that CentOS 7 defaults to an 
> xfs filesystem (which
> I'm using). I'm pretty sure that you have to use xfsdump for that.
>
> I posted on the CentOS list as well. Here's what they said
> https://lists.centos.org/pipermail/centos/2015-September/155046.html (note 
> there are a couple of
> unsigned test packages referenced there as well:
> https://lists.centos.org/pipermail/centos/2015-March/150446.html)
Are there any plans to support the xfs filesystem with xfsdump? Has anyone 
encountered this before?



signature.asc
Description: OpenPGP digital signature

Re: ERROR: client: [DUMP program not available]

[Tom Robinson]

On 24/09/15 08:23, Tom Robinson wrote:
> Hi Paul,
>
> Thanks and, yes, that was it. The community package to which you refer works.

Well, I spoke too soon. I neglected to remember that CentOS 7 defaults to an 
xfs filesystem (which
I'm using). I'm pretty sure that you have to use xfsdump for that.

I posted on the CentOS list as well. Here's what they said
https://lists.centos.org/pipermail/centos/2015-September/155046.html (note 
there are a couple of
unsigned test packages referenced there as well:
https://lists.centos.org/pipermail/centos/2015-March/150446.html)



signature.asc
Description: OpenPGP digital signature

Re: Upgrade woes and eternal hanging of dumps

[Seann]

On 9/21/2015 11:36 AM, Debra S Baddorf wrote:

YES!I agree with the first and third of these tidbits.  I just couldn’t 
remember them.  I’ve had issues with both of them.  Including the tricky 
firewall timeout part,  in Idea Three.

Here’s hoping you have a network person who can add some skills or ideas at 
that level.   Or,  just don’t do client estimates,  as in the first suggested 
fix.

I think we had to allow trusted clients to initiate their OWN connections back 
to the server  (via a firewall rule),  so that they could still talk to the 
server even after that server-created conversation had timed out.   That might 
count as fix #3,  but it takes firewall skills.   That might be a slightly 
different problem/situation  (it sounds a little different)  but I think it’s 
in this same category, somewhere.  Network savvy people, can you translate my 
“generic English” description into what we actually did?
Deb Baddorf,  Fermilab


On Sep 21, 2015, at 10:25 AM, Joi L. Ellis  wrote:


I've just read through the long thread prompted by this particular post.  I'd 
like to offer a few points I didn't see mentioned before...

Idea one: You upgraded from 2.5 to 3.3.  2.5 amdump only spoke UDP with a 'bsd' 
auth protocol, so that was the only action available.  Thus, inetd.conf didn't 
specify an -auth=bsd parameter.  3.3 defaults to -auth=bsdtcp if you don't 
provide it.  Does your new configuration specify that those clients must be 
reached with -auth=bsd from the new server, rather than the server's new 
default of -auth=bsdctp?

Idea two: If any of the involved machines are running iptables or ufw 
firewalls, verify the new configuration is still loading the correct kernel 
modules. At one point the /etc/default/ufw.conf file named kernel modules 
incorrectly after an upgrade, and/or the nf_conntrack_amanda module itself went 
missing.  (Some kernels change the name of this module, usually it's the first 
two characters.)  The symptom here is that amcheck thinks everything is fine, 
yet the actual amdump process fails because the  UDP control conversation 
between the server and the client is allowed, but the TCP data stream amdump 
uses with -auth=bsdtcp is blocked.

Idea Three: I run an Amanda 3.3.3 server, and I have experienced a similar 
problem to your own.  I've tried posting about it here in the past and got null 
response, so I gave up asking for help and figured out my own workarounds.

My amanda server is behind a corporate firewall, and some of the clients are in 
the DMZ, outside the firewall... and they are running amanda 2.5 due to the age 
of the client hosts.   I've had repeated issues with the corporate firewall 
interfering with the planner.

The issue I see is that the amanda server planner fires off a UDP "connection" 
to the client, asking the client to provide estimates.  The client does so... BUT.  That 
blasted firewall has created a dynamic NAT rule that will allow the client to send back 
its UDP response.  IF the client's response doesn't appear before the NAT rule expires, 
the planner falls into a permanent wait state, waiting for a UDP response that will never 
arrive because the firewall has blocked it.  The client has no idea it failed, and its 
logs look entirely normal.

If you dig into the server's logs, you will probably find TIMEOUT errors in the 
logs from the planner.  I don't have any recent logs that illustrate this 
error, so I can't quote an example.

I worked around this in two ways (varies with the client situation:)

  *) tell amanda to not use the client to create the estimate at all
  *) adjust the NAT timeout rules on the firewall to extend the timeout.  As I 
recall, it was initially set to 120 seconds.  We moved it up to 300 seconds at 
one point, but then began to experience issues with the firewall filling memory 
tables because rules weren't timing out fast enough.

As I see it, the planner makes the (unsafe) assumption that IF its initial 
request-an-estimate packets traveled properly, the response will always do so.  
If there is a firewall involved, the response might get lost, yet the planner 
will sit there forever, twiddling its thumbs and not backing up anything, until 
it receives the missing estimates package back from the client.

To summarize, I suspect that the move from 2.5's UDP-only communication style 
to 3.3's default TCP-only style has broken something in your environment that 
you've overlooked.  Either the server, the clients (or both) or a firewall 
(either an external network firewall, or a kernel firewall on one of the hosts 
involved) are breaking your planner.  I've experienced very similar symptoms 
after version upgades.

(And yes, I've seen my issues disappear when jobs are run manually, yet still 
fail when run over night.  Manual tests don't trigger the firewall issues 
because the windows I have open the to the client and server keep the darn 
firewall from timing out the dynamic NAT