Re: Are you running snapshots?

[Gene Heskett]

On Wednesday 12 August 2009, Dustin J. Mitchell wrote:
>On Wed, Aug 12, 2009 at 7:13 AM, Jean-Louis
>
>Martineau wrote:
>> There is no new snapshot because there is no commit since 20090805.
>
>Ah, so.  I've been off my game!
>
>Dustin

And I'd advise that one should not fix that which is not broken. :)

20090805 is working well here.  Of course I probably do not exercise it to the 
fullest by any means.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
The NRA is offering FREE Associate memberships to anyone who wants them.


Equal bytes for women.

Re: Are you running snapshots?

[Dustin J. Mitchell]

On Wed, Aug 12, 2009 at 7:13 AM, Jean-Louis
Martineau wrote:
> There is no new snapshot because there is no commit since 20090805.

Ah, so.  I've been off my game!

Dustin

-- 
Open Source Storage Engineer
http://www.zmanda.com

Re: Are you running snapshots?

[Jean-Louis Martineau]

Dustin J. Mitchell wrote:

Hmm, well one problem is that that's a snapshot of the the 2.6.1
branch, which is pretty stable at this point.  The other problem,
which Jean-Louis might be able to address, is that it's now almost
20090812, so we should probably have a new build installed by now.
  

There is no new snapshot because there is no commit since 20090805.

Jean-Louis

Re: Are you running snapshots?

[Gene Heskett]

On Tuesday 11 August 2009, Dustin J. Mitchell wrote:
>On Tue, Aug 11, 2009 at 12:56 PM, Gene Heskett wrote:
>> I have been following the 2.6.2 snapshots, usually within a day or so, for
>> several months.  You've heard about it when I have problems, which isn't
>> often.
>
>Thanks, Gene -- this has been a great help!
>
>Dustin

You are welcome Dustin.  But understand I do this with the image of the canary
in the coal mine in mind.  If my canary dies, its time to yelp.  Its been 
pretty healthy of late so I don't have anything to fuss about other than tars
penchant for triggering an Oopsen on the estimate run of the first amanda
invocation after a reboot.

Just in case you haven't seen that yet, here it is so you can pass judgment 
and/or point a finger:
===
Aug  5 01:15:01 coyote xinetd[2259]: START: amanda pid=19290 
from=:::192.168.71.3
Aug  5 01:16:16 coyote kernel: [ 8621.118387] BUG: Bad page state in process 
tar  pfn:a1193
Aug  5 01:16:16 coyote kernel: [ 8621.118394] page:c28fc260 flags:80004000 
count:0 mapcount:0 mapping:(null) index:0
Aug  5 01:16:16 coyote kernel: [ 8621.118398] Pid: 19517, comm: tar Not tainted 
2.6.31-rc5 #3
Aug  5 01:16:16 coyote kernel: [ 8621.118400] Call Trace:
Aug  5 01:16:16 coyote kernel: [ 8621.118408]  [] ? printk+0x23/0x40
Aug  5 01:16:16 coyote kernel: [ 8621.118414]  [] bad_page+0xcf/0x150
Aug  5 01:16:16 coyote kernel: [ 8621.118417]  [] 
get_page_from_freelist+0x37d/0x480
Aug  5 01:16:16 coyote kernel: [ 8621.118421]  [] ? 
add_to_page_cache_lru+0x84/0x90
Aug  5 01:16:16 coyote kernel: [ 8621.118424]  [] 
__alloc_pages_nodemask+0xdf/0x520
Aug  5 01:16:16 coyote kernel: [ 8621.118429]  [] ? 
hrtimer_interrupt+0xdd/0x250
Aug  5 01:16:16 coyote kernel: [ 8621.118432]  [] 
__do_page_cache_readahead+0x104/0x220
Aug  5 01:16:16 coyote kernel: [ 8621.118436]  [] ra_submit+0x34/0x50
Aug  5 01:16:16 coyote kernel: [ 8621.118438]  [] 
ondemand_readahead+0x120/0x240
Aug  5 01:16:16 coyote kernel: [ 8621.118441]  [] 
page_cache_async_readahead+0x9c/0xb0
Aug  5 01:16:16 coyote kernel: [ 8621.118445]  [] 
generic_file_aio_read+0x33c/0x6a0
Aug  5 01:16:16 coyote kernel: [ 8621.118468]  [] 
do_sync_read+0xe9/0x140
Aug  5 01:16:16 coyote kernel: [ 8621.118472]  [] ? 
autoremove_wake_function+0x0/0x60
Aug  5 01:16:16 coyote kernel: [ 8621.118477]  [] ? 
security_file_permission+0x1e/0x40
Aug  5 01:16:16 coyote kernel: [ 8621.118479]  [] ? 
rw_verify_area+0x60/0xe0
Aug  5 01:16:16 coyote kernel: [ 8621.118483]  [] vfs_read+0xb7/0x180
Aug  5 01:16:16 coyote kernel: [ 8621.118485]  [] ? 
do_sync_read+0x0/0x140
Aug  5 01:16:16 coyote kernel: [ 8621.118488]  [] sys_read+0x58/0xa0
Aug  5 01:16:16 coyote kernel: [ 8621.118491]  [] 
sysenter_do_call+0x12/0x22
Aug  5 01:16:16 coyote kernel: [ 8621.118494] Disabling lock debugging due to 
kernel taint
Aug  5 01:21:01 coyote xinetd[2259]: EXIT: amanda status=0 pid=19290 
duration=360(sec)
=
that was I believe the estimate run.

And once it has that out of its system, no repeats, uptime is now 4d 13:11 on
this kernel.  So its your call, I've been pointing my finger at the kernel.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
The NRA is offering FREE Associate memberships to anyone who wants them.


All warranty and guarantee clauses become null and void upon payment of invoice.

Re: Are you running snapshots?

[Dustin J. Mitchell]

On Tue, Aug 11, 2009 at 4:48 PM, Jean-Francois
Malouin wrote:
> Can you elaborate on the new chg-robot: I just downloaded the latest
> snapshot for 2.6.1 (2.6.1p1-20090805) and I can't find anything
> related to that.

Hmm, well one problem is that that's a snapshot of the the 2.6.1
branch, which is pretty stable at this point.  The other problem,
which Jean-Louis might be able to address, is that it's now almost
20090812, so we should probably have a new build installed by now.
That said, the commit of chg-robot (perl/Amanda/Changer/robot.pm) was
on 8/3, so the 2.6.2alpha-20090805 snapshot should do the trick for
you.

> In the next few days I intend to toy with this as I have a new library
> not in yet production.

That would be great!  I've tested this manually on the changers I have
access to, but its unit tests use "mock" versions of mt and mtx, which
might obscure some bugs that will appear on exposure to more real
hardware.  I've been toying with the idea of running the unit tests
against a real changer library, but there are just too many variables
and too many possibilities for false failures involved.

> Thanks for the good work!

And to you, in advance!

Dustin

-- 
Open Source Storage Engineer
http://www.zmanda.com

Re: Are you running snapshots?

[Jean-Francois Malouin]

* Dustin J. Mitchell  [20090811 10:34]:
> I'm curious to know who out there is running the daily Amanda snapshots.
> 
> A *lot* of new code has gone in since the 2.6.1 release, and I'd like
> to see it tested now, rather than waiting until the beta process
> begins.  We've rewritten the taper to use the transfer architecture;
> rewitten all uses of the old taperscan algorithm to use the simpler
> and more explicit Amanda::Taper::Scan::traditional; rewritten several
> other applications to use the new Changer API; and added a new
> chg-robot to replace the crusty old chg-zd-mtx.  I'm working on

Can you elaborate on the new chg-robot: I just downloaded the latest
snapshot for 2.6.1 (2.6.1p1-20090805) and I can't find anything
related to that.

In the next few days I intend to toy with this as I have a new library
not in yet production.

Thanks for the good work!
jf

> rewriting the restore/recover applications now.  The idea is to
> completely jettison the old Changer API before the next relase.
> 
> So, please let me know if the snapshots are working for you!
> 
> Dustin
> 
> -- 
> Open Source Storage Engineer
> http://www.zmanda.com

Re: Are you running snapshots?

[Dustin J. Mitchell]

On Tue, Aug 11, 2009 at 12:56 PM, Gene Heskett wrote:
> I have been following the 2.6.2 snapshots, usually within a day or so, for
> several months.  You've heard about it when I have problems, which isn't
> often.

Thanks, Gene -- this has been a great help!

Dustin

-- 
Open Source Storage Engineer
http://www.zmanda.com

Re: Are you running snapshots?

[Gene Heskett]

On Tuesday 11 August 2009, Dustin J. Mitchell wrote:
>I'm curious to know who out there is running the daily Amanda snapshots.
>
>A *lot* of new code has gone in since the 2.6.1 release, and I'd like
>to see it tested now, rather than waiting until the beta process
>begins.  We've rewritten the taper to use the transfer architecture;
>rewitten all uses of the old taperscan algorithm to use the simpler
>and more explicit Amanda::Taper::Scan::traditional; rewritten several
>other applications to use the new Changer API; and added a new
>chg-robot to replace the crusty old chg-zd-mtx.  I'm working on
>rewriting the restore/recover applications now.  The idea is to
>completely jettison the old Changer API before the next relase.
>
>So, please let me know if the snapshots are working for you!
>
>Dustin

I have been following the 2.6.2 snapshots, usually within a day or so, for 
several months.  You've heard about it when I have problems, which isn't 
often.

Currently running the 2.6.2-20090805 snapshot.  The only problem is an Ooopsen 
caused by tar, in the first minute thirty of the first run of amanda after a 
reboot, an Ooopsen that is apparently 100% benign as it never repeats, and 
amanda is not aware of it, the backup proceeds normally.

There is a regression filed against kernel 2.6.31-rc5 about this, but whose bz 
number I do not recall ATM.  Ends in 899 IIRC.  Current uptime on that kernel 
is 4d, 1:23.

I have been reducing the size of the vtape slowly, and the scheduler seems to 
be coming into a balance of about 85% usage, so I just reduced it another few 
percent, from 16GB to 15GB.  Drive usage is still just under 50% on a 1TB 
drive so I could add more vtapes, but see no real reason to, 30 seems 
sufficient.

Is it ready for prime time?  My guess is that it is, at least for me, in my 
environment.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
The NRA is offering FREE Associate memberships to anyone who wants them.


We don't care how they do it in New York.

Re: Are you running snapshots?

[Alan Pearson]

Good points and I do appreciate the effort that is put into AMANDA and its
backwards compatibility.
I guess testing the client is easy, and something I will try to do over
the next few weeks and report the results, but changing the server side..
well that scares me :)

:)

-- 
AlanP

On Tue, August 11, 2009 4:48 pm, Dustin J. Mitchell wrote:
> On Tue, Aug 11, 2009 at 10:47 AM, Alan Pearson
> wrote:
>> You have to appreciate a lot of people have AMANDA in production
>> environments and it is working. We are very reluctant to change it in
>> case
>> it breaks, we don't want the overhead of having to fix it.
>
> Absolutely -- I hope I did not imply that everyone, or even most
> people, should be running snapshots!  But hopefully at least a few
> people can spare some resources to test things out and alert us to any
> problems.
>
>> I am especially nervous to change since AMANDA has changed so much (read
>> -
>> the change to PERL) and is no longer a straight upgrade with bugfixes
>> and
>> new features
>
> We're working double-time to ensure backward compatibility, and any
> failing on that point is a bug.  But certainly backups are often a
> set-it-and-forget-it kind of technology, and frequent upgrades can be
> disruptive, even if they are smooth.
>
>> I personally think this is a failing of the AMANDA team to realise this,
>> and to see that a lot of sys admins are reluctant to upgrade for these
>> reasons.
>
> I realize that Amanda is working great *right now* for a number of
> people, but if the project is to remain relevant, it must serve the
> needs of today's new installations, while remaining fully compatibile
> with existing installations.
>
> That's a lot to ask, and it's not easy.  I think we deserve some
> credit for our success so far, and some support for our continued
> committment to backward compatibility.  Which brings me back to the
> original topic: any and all assistance with testing recent versions of
> Amanda in different situations is extraordinarily helpful to ensuring
> backward compatibility.
>
> For example, Alan, it would be great to hear from you that the
> Kerberos authentication continues to function in the latest snapshots,
> particularly since it is not something to which we can apply unit
> tests (we have almost 2000 unit tests at this point, by the way).
> That only has to mean setting up Amanda on a throwaway machine or VM,
> running amservice to verify the authentication, and dropping a note to
> amanda-users@amanda.org with the results.
>
> The snapshots are here:
>   http://www.zmanda.com/community-builds.php
> and to be clear: please do *not* run snapshots in production!
>
> Dustin
>
> --
> Open Source Storage Engineer
> http://www.zmanda.com
>

Re: Are you running snapshots?

[Alan Pearson]

Hi Dustin

You have to appreciate a lot of people have AMANDA in production
environments and it is working. We are very reluctant to change it in case
it breaks, we don't want the overhead of having to fix it.

Example we are deploying a 2.5.2 snapshot that has fixes in it that we
raised.

Speaking personally, I will not upgrade my AMANDA environment until I
absolutely have to, as it took quite a bit of effort to get it working
with our changer, kerberos auth and environment.

I am especially nervous to change since AMANDA has changed so much (read -
the change to PERL) and is no longer a straight upgrade with bugfixes and
new features
I personally think this is a failing of the AMANDA team to realise this,
and to see that a lot of sys admins are reluctant to upgrade for these
reasons.


Just my 2c

AlanP

-- 
AlanP

On Tue, August 11, 2009 2:57 pm, Dustin J. Mitchell wrote:
> I'm curious to know who out there is running the daily Amanda snapshots.
>
> A *lot* of new code has gone in since the 2.6.1 release, and I'd like
> to see it tested now, rather than waiting until the beta process
> begins.  We've rewritten the taper to use the transfer architecture;
> rewitten all uses of the old taperscan algorithm to use the simpler
> and more explicit Amanda::Taper::Scan::traditional; rewritten several
> other applications to use the new Changer API; and added a new
> chg-robot to replace the crusty old chg-zd-mtx.  I'm working on
> rewriting the restore/recover applications now.  The idea is to
> completely jettison the old Changer API before the next relase.
>
> So, please let me know if the snapshots are working for you!
>
> Dustin
>
> --
> Open Source Storage Engineer
> http://www.zmanda.com
>

Re: Are you running snapshots?

[Dustin J. Mitchell]

On Tue, Aug 11, 2009 at 10:47 AM, Alan Pearson wrote:
> You have to appreciate a lot of people have AMANDA in production
> environments and it is working. We are very reluctant to change it in case
> it breaks, we don't want the overhead of having to fix it.

Absolutely -- I hope I did not imply that everyone, or even most
people, should be running snapshots!  But hopefully at least a few
people can spare some resources to test things out and alert us to any
problems.

> I am especially nervous to change since AMANDA has changed so much (read -
> the change to PERL) and is no longer a straight upgrade with bugfixes and
> new features

We're working double-time to ensure backward compatibility, and any
failing on that point is a bug.  But certainly backups are often a
set-it-and-forget-it kind of technology, and frequent upgrades can be
disruptive, even if they are smooth.

> I personally think this is a failing of the AMANDA team to realise this,
> and to see that a lot of sys admins are reluctant to upgrade for these
> reasons.

I realize that Amanda is working great *right now* for a number of
people, but if the project is to remain relevant, it must serve the
needs of today's new installations, while remaining fully compatibile
with existing installations.

That's a lot to ask, and it's not easy.  I think we deserve some
credit for our success so far, and some support for our continued
committment to backward compatibility.  Which brings me back to the
original topic: any and all assistance with testing recent versions of
Amanda in different situations is extraordinarily helpful to ensuring
backward compatibility.

For example, Alan, it would be great to hear from you that the
Kerberos authentication continues to function in the latest snapshots,
particularly since it is not something to which we can apply unit
tests (we have almost 2000 unit tests at this point, by the way).
That only has to mean setting up Amanda on a throwaway machine or VM,
running amservice to verify the authentication, and dropping a note to
amanda-users@amanda.org with the results.

The snapshots are here:
  http://www.zmanda.com/community-builds.php
and to be clear: please do *not* run snapshots in production!

Dustin

-- 
Open Source Storage Engineer
http://www.zmanda.com

Re: Are you running snapshots?

[Dustin J. Mitchell]

On Tue, Aug 11, 2009 at 10:43 AM, Brian Cuttler wrote:
> Snapshots are great - but not the solution to all problems,
> files that are open when the snapshots are created are 'open'
> when you back them up (not everyone here seems to understand this).

Sorry -- I meant the daily snapshots of the latest and greatest Amanda
code.  I should have made that clear :)

Dustin

-- 
Open Source Storage Engineer
http://www.zmanda.com

Re: Are you running snapshots?

[Brian Cuttler]

Our shop is running with zfs-snapshots.
We don't/haven't attempted ufs-snapshots.

Snapshots are great - but not the solution to all problems,
files that are open when the snapshots are created are 'open'
when you back them up (not everyone here seems to understand this).

On Tue, Aug 11, 2009 at 09:57:21AM -0400, Dustin J. Mitchell wrote:
> I'm curious to know who out there is running the daily Amanda snapshots.
> 
> A *lot* of new code has gone in since the 2.6.1 release, and I'd like
> to see it tested now, rather than waiting until the beta process
> begins.  We've rewritten the taper to use the transfer architecture;
> rewitten all uses of the old taperscan algorithm to use the simpler
> and more explicit Amanda::Taper::Scan::traditional; rewritten several
> other applications to use the new Changer API; and added a new
> chg-robot to replace the crusty old chg-zd-mtx.  I'm working on
> rewriting the restore/recover applications now.  The idea is to
> completely jettison the old Changer API before the next relase.
> 
> So, please let me know if the snapshots are working for you!
> 
> Dustin
> 
> -- 
> Open Source Storage Engineer
> http://www.zmanda.com
---
   Brian R Cuttler brian.cutt...@wadsworth.org
   Computer Systems Support(v) 518 486-1697
   Wadsworth Center(f) 518 473-6384
   NYS Department of HealthHelp Desk 518 473-0773



IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure.  It
is intended only for the addressee.  If you received this in error or
from someone who was not authorized to send it to you, please do not
distribute, copy or use it or any attachments.  Please notify the
sender immediately by reply e-mail and delete this from your
system. Thank you for your cooperation.