Re: Meeting Minutes : Amanda Open Source Community

[J Chapman Flack]

On 3/19/19 2:31 PM, Gene Heskett wrote:

> Here's what I have ATM.
> ...
> amanda-4.0.0alpha.svn.4700.tar.gz
> ...
> amanda-4.0.0alpha.svn.5199.tar.gz

Did those come from the SVN repo at https://sourceforge.net/p/amanda ?
Or is there another I've overlooked?

The one at sourceforge has a latest revision r7719 from 14 Feb 2019.

The mirror at https://github.com/zmanda/amanda seems to stop at
git commit b2fd140, which corresponds to svn r7714 about a year
earlier. Then apparently the updates to git stopped happening.

The later commits I see in sourceforge svn are:
7715 (branch 3_5) for filtering arguments to GNU tar
7716 (branch 3_5) for security review comments fixes
7717 (creates branch 3_5_Debug from 3_5)
7718 (branch 3_5_Debug) changes to s3 device
7719 (branch 3_5_Debug) adds debug logging for s3

> Start a new github entry as amanda-4-alpha.

Surely the way to go would be to fork the existing github repo
(for preservation of all existing history and commit IDs) and
catch it up (via git-svn) to any history in the svn repo that
isn't mirrored over yet?

Or is the 4.0.0alpha stuff coming from some completely different
repository?

-Chap

Re: Reusable

[J Chapman Flack]

On 11/26/18 2:32 PM, J Chapman Flack wrote:
> Jean-Louis mentioned the amraw application as one that can be used on
> a single file:  http://wiki.zmanda.com/man/3.5.1/amraw.8.html
> 
> In the pending pull request #11, there are some others as well:
> 
> amooraw:... amgrowingfile:... amgrowingzip:...
I've realized I gave the wrong PR number here, it is pull request
#78. Sorry.

  https://github.com/zmanda/amanda/pull/78

I was looking at the number saying it's currently the 11th open PR.

-Chap

Re: Reusable

[J Chapman Flack]

On 11/21/18 2:55 PM, Chris Miller wrote:
> AMANDA can't backup a file; only directories. Is there any way around
> that? Suppose I need to backup a large file in a directory with many
> other large files, so backing up the complete directory or moving /
> copying my target would be inconvenient, not to mention
> administratively cumbersome when it comes to restoring. 

Jean-Louis mentioned the amraw application as one that can be used on
a single file:  http://wiki.zmanda.com/man/3.5.1/amraw.8.html

In the pending pull request #11, there are some others as well:

amooraw: this is just the same as amraw but implemented in the OO style
made possible in PR#11, to show how the development is simplified. Like
amraw, it only supports full backups.

amgrowingfile: this is specific to the case where the single file is
something that grows very large and only by appending (think audit or
log files, etc). It can do efficient incremental backups of the single
file (an increment is just what has been appended since the last one).
A level 0 has to be forced any time the file is rewritten any other
way than appending.

amgrowingzip: like amgrowingfile but for the special case where the
single file is a large zip archive that is only grown by appending
new members at the end. It also supports increments. A level 0 must
be forced any time the file is modified any other way than adding new
zip members at the end.

In the preliminary man pages with the pull request, amooraw is on
page 12, amgrowingfile on page 9, and amgrowingzip on page 10.

https://github.com/zmanda/amanda/files/1265069/AppScriptWithAbstractClasses.pdf

-Chap

Re: Clients that return something else

[J Chapman Flack]

On 11/15/18 2:49 PM, Chris Hoogendyk wrote:
> This
> 
> is a bit dated, but may give you an idea of how to attack it. However,
> it could be that with the api work that Dustin did there is a better way
> of doing this with the newer versions of Amanda.

Yes, it sounded to me as if this was a question for the
Application / Script API. Amanda has very flexible support for
letting you write a script that can turn whatever-you've-got
into a stream of bytes in any way that makes sense, and turn a
stream of bytes back into whatever-it's-supposed-to-be, and Amanda
will use that for backups, and store the byte streams away, and
amrecover will fetch the bytes back and use your app/script to
turn them back into whatever-it's-supposed-to-be.

The API in the current released versions has a kind of steep
learning curve. There is an open pull request aimed at making it
much easier to use:  https://github.com/zmanda/amanda/pull/78

The pull request also includes several more example apps and
scripts, for things like backing up filesystems on LVM using
snapshots, making consistent hot backups of Subversion repositories
or 389 Directory Server instances, another approach to ZFS backups
where some other process is creating regular snapshots and Amanda
just works from those, etc. The examples should give a good sense
of what can be built when the API is easy enough to work with.

The docs for those examples are here:
https://github.com/zmanda/amanda/files/1265069/AppScriptWithAbstractClasses.pdf

-Chap

Re: sharing scripts

[J Chapman Flack]

On 11/8/18 12:27 PM, Gene Heskett wrote:

> I certainly don't have any objections, and I wasn't aware there were 
> anywhere near that many choices.  Can you PM me a list?

In this instance, PM stands for "public message" ... they're all
listed under Insights / Forks on the zmanda/amanda GitHub page:

https://github.com/zmanda/amanda/network/members

-Chap

Re: Meeting Minutes : Amanda Open Source Community

[J Chapman Flack]

On 10/30/18 5:58 AM, Stefan G. Weichinger wrote:

> Provide an issue tracker etc ASAP for users to be able to contribute
> feature requests and patches etc

As there is already https://github.com/zmanda/amanda with an issue
tracker, wiki, pull requests, etc., and the full source history thanks
to mirroring svn ...

Would it be feasible to just pick a date and retire the svn repo
and declare the github one to be the locus of development?

-Chap

Re: Meeting Minutes : Amanda Open Source Community

[J Chapman Flack]

On 10/30/18 10:13 AM, Stefan G. Weichinger wrote:
> Am 30.10.18 um 15:06 schrieb J Chapman Flack:
>> On 10/30/18 5:58 AM, Stefan G. Weichinger wrote:
>>
>> Would it be feasible to just pick a date and retire the svn repo
>> and declare the github one to be the locus of development?
> 
> This has to be decided by the owner of Amanda, BETSOL, IMO.
> 
> And personally I am waiting for decisions and announcements like that.

I agree. Just suggesting an easily-implemented possibility 

Re: Zmanda acquired from Carbonite by BETSOL -- future of Amanda development?

[J Chapman Flack]

On 10/16/2018 05:38 AM, Stefan G. Weichinger wrote:
> Am 15.10.18 um 17:17 schrieb Ashwin Krishna:
>> Let me go back and check with the team.  This has to be sent out.
> 
> I am curious already, and I assume other amanda-users as well ... ?
Yes.

Re: Zmanda acquired from Carbonite by BETSOL -- future of Amanda development?

[J Chapman Flack]

On 09/27/2018 06:24 PM, Todd M. Kover wrote:

> The web stuff is hosted at sourceforge and anyone with commit access
> there is able to update it/maintain it.  The code transitioned to
> github, but the website stayed behind.

That's interesting ... there's still this wiki page
http://wiki.zmanda.com/index.php/Fork_Amanda_on_Github
saying svn is "the reference source repository" and
github is "never far behind". Is that still accurate, or
is it out of date and github is now authoritative?

I think both Jean-Louis and Ashwin have mentioned 'svn write
access' in the past day or so.

-Chap

Re: Zmanda acquired from Carbonite by BETSOL -- future of Amanda development?

[J Chapman Flack]

On 09/27/2018 11:05 AM, Ashwin Krishna wrote:

> I have reached out to members of -- owner-amanda-users group,
> postmas...@amanda.org , webmas...@amanda.org in the past offering
> our help and support to maintain amanda community. 

It's possible I've never known this -- who are the people who
maintain the amanda.org website and mailing lists, or receive
postmaster/webmaster/-owner mail?

Have they been Carbonite people in the past, or volunteers from
the larger community? Are they credited somewhere?

-Chap

Re: Zmanda acquired from Carbonite by BETSOL -- future of Amanda development?

[J Chapman Flack]

On 09/26/2018 02:13 PM, J Chapman Flack wrote:

> Does anyone know what became of Jean-Louis Martineau

I see that my last email crossed with one from him. :)

Re: Zmanda acquired from Carbonite by BETSOL -- future of Amanda development?

[J Chapman Flack]

On 09/26/2018 01:35 PM, Chris Nighswonger wrote:
> On Wed, Sep 26, 2018 at 1:31 PM Gene Heskett  wrote:
>> Fork it if someone has a long term interest in seeing a good, long term
>> backup solution keep sucking air regularly.
>>
>> Said by a 20 year user of amanda.
> 
> +1 to forking.

Is there some one organization among the long-term users of amanda
that would be a natural choice as keeper of a community fork?
Should a dedicated organization be created for the purpose?

Does anyone know what became of Jean-Louis Martineau in the sale
to BETSOL, or whether he still has interest and time for the project?

I would be happy to contribute to a forked community amanda (or
whatever it might be called, if "amanda" trademark rights are in
question). I still have a batch of contributions I was working with
Jean-Louis to get pushed a year or so ago, but it seems we both got
pulled onto other things.

However, I don't have the bandwidth available to be a maintainer
of a fork

-Chap

Re: Oddity with the disklist

[J Chapman Flack]

Where did the 091 come from? I don't see it in the first (working) example.

-Chap

On 06/01/18 12:09, Dirk-Willem van Gulik wrote:
> I’ve got a bit of an oddity with the disklist - suspect it is really trivial; 
> but am not seeing it.
> 
> This works splendidly and as expected:
> 
>   ...
>   .webweaving.orgtank/jail/webhost 
> openssl-client-encrypt-best-zfssnapshot-jail
> 
> But this (with whitespace reduced to just space and a \n straight after every 
> line, no space in between):
> 
>   .webweaving.org  tank/jail/webhost091 { 
>   
>   openssl-client-encrypt-best-zfssnapshot-jail
>   exclude append "./usr/local/www/logs"
>   }
> 
> Gives me the error:
> 
>   ERROR: .webweaving.org: [can not stat tank/jail/webhost091: No such 
> file or directory]
> 
> And bringing it back to just:
> 
>   .webweaving.org  tank/jail/webhost091 { 
>   
>   openssl-client-encrypt-best-zfssnapshot-jail
>   }
> 
> gives me the same error. This is on Amanada 3.3.9, stock FreeBSD install 
> against a stock FreeBSD-11.1 remote host with zfs #28.
> 
> What am I missing. Must be trivial !
> 
> Thanks,
> 
> Dw.
> 
> 

Re: Databases

[J Chapman Flack]

On Jan 5, 2018, at 2:22 PM, Chris Miller  wrote:
> I can't find any explicit guidance for databases, so I assume that
> I do any database preparation in the cron job before I invoke amdump,
> and then tell Amanda to prune the database. Am I right?  OR, is there
> an agent for the common databases, like MySQL, and PostgresSQL?

There's one for PostgreSQL, described here:

http://wiki.zmanda.com/man/3.5.1/ampgsql.8.html

It's one example of an Amanda 'application', which is essentially
a plug-in for Amanda to back up a certain special kind of stuff.

If you can write a little program (in Perl or another language)
that, roughly, knows how to read your special kind of stuff and
send it out a pipe as a stream of bytes, or later get that stream
of bytes piped back to it and write out the recovered stuff, you
can write an application specific to another database, or other
specialized application you want to back up. This extensibility
is an Amanda strength (if a bit underutilized; there is a pull
request pending right now to make it easier to do).

In the case of a DBMS like PostgreSQL, there is a lot of latitude
for deciding what you even mean by "taking a backup", "pruning",
"incremental backup", and so on. The existing ampgsql application
implements one particular way of thinking about that. If a different
way would fit your setup better, it could be implemented as another
application (maybe just ampgsql slightly changed, or perhaps very
different).

-Chap

Re: make DLE wait for a job

[J Chapman Flack]

On 02/13/2017 02:47 PM, Debra S Baddorf wrote:

>> If that would work per DLE it would be nice.
>> Something like a dumptype-parameter ;-)

> Yes, that’s a built in thing,  but yes it affects the whole dump run.

I might suggest an addition to the Script API so that a
pre-dle-estimate or pre-dle-backup script can report back
a special status (like "try again in N minutes").

Then it'd be easy to just write a script for a particular DLE
that will check whatever is appropriate, and decide when to
proceed.

-Chap

Re: Software vs: Hardware compression

[J Chapman Flack]

On 10/28/2016 02:37 PM, Chris Hoogendyk wrote:

> all of the data is being compressed, and the compression is significant,
> but it has become the bottleneck. Top shows multiple of Amanda's gz
> processes at the top of the list all day.

In your setup, are there enough DLEs to compress that Amanda can keep
all your cores busy with gz processes working on different DLEs?

Or do you have some cores underutilized, that you could maybe bring
into the game by using pigz instead of gz?

Chapman Flack

Re: Software vs: Hardware compression

[J Chapman Flack]

On 10/28/2016 02:37 PM, Chris Hoogendyk wrote:
> It also knows what a particular DLE can be compressed to based
> on the history. If the tape drive is doing the compression, then it is a
> black box. Amanda doesn't know what the DLE got compressed to, and it
> doesn't know how that relates to the capacity of the tape. That makes
> planning more difficult.

This might be a good place to plug an idea I floated back in January:
http://marc.info/?l=amanda-hackers=145312885902912=2

... it turns out in the SCSI standard for tape drive commands,
there are ways to ask the drive, after writing each fileset,
"hey drive! what did that last set compress to?"

I haven't had time to pursue it ... I think the part I could probably
do without help is teaching the tape driver to ask the drive for the
statistics. It might take a more seasoned Amanda hacker to work out
how to get those numbers stored in the history, similarly to what
happens with software compression, so they can be used by the planner
later.

It still does strike me as worth exploring

Chapman Flack