Re: UFS Snapshots and iowait

[krad]

On 23 November 2010 19:49, Kurt Lidl  wrote:

> On Tue, Nov 23, 2010 at 10:38:31AM -0800, Chris St Denis wrote:
> > Is this just due to the very high io bandwidth usage associated with
> > making a snapshot, or does the creation of this snapshot completely
> > block IO writes for around 5 minutes?
>
> It blocks updates to the filesystem while during part of the
> snapshot process.
>
> See the comments in /usr/src/sys/ufs/ffs/ffs_snapshot.c
>
> I found using UFS snapshots on a production fileserver untenable
> during normal working hours.  I have a backup fileserver that I
> rsync the files to, and then use the UFS snapshots there.
>
> > Any suggested workarounds? I already bumped up the number of Apache
> > slots to 166% but it looks like I would have to increase the number much
> > more to use that as a primary solution.
>
> Use ZFS.  The way snapshots work there, they are nearly instantanous
> to create, and you are not limited to 20 snapshots per filesystem.
>
> -Kurt
> ___
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
>

I can testify zfs snapshots are very usable, as we use it to backup our
mysql and oracle databases. Issue a write lock, flush, snap, remove lock,
backup snapshot All takes a few seconds and is fairly seamless
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Re: getting a list of open files versus PID nos.?

[krad]

On 9 December 2010 00:20, Andriy Gapon  wrote:

> on 09/12/2010 01:47 Matthew Fleming said the following:
> > On Wed, Dec 8, 2010 at 3:01 PM, Bruce Cran  wrote:
> >> On Wed, 8 Dec 2010 14:54:57 -0800
> >> Matthew Fleming  wrote:
> >>
> >>> This is what lsof is for.  I believe there's one in ports, but I have
> >>> never tried it.
> >>
> >> Is there any advantage to using lsof instead of fstat(1) (fstat -p pid)?
> >
> > I believe that lsof reports on all open files by all processes,
> > whereas fstat will only report on a specific provided pid.
>
> Just try running fstat without any options.
> Or procstat -a -f.
>
> --
> Andriy Gapon
> ___
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
>


not sure if dtrace is ready for it on freebsd yet, but it certainly can do
it on solaris
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Re: Freebsd on the sun x4440

[krad]

On 17 May 2012 01:34, Mark Saad  wrote:

> All
>  I am setting up 3 sun x4440 with freebsd 7.3 amd64 . The severs have 4x
> 4-core opterons and 128G of ram each . Once the servers boot they are a good
> fit for what we are doing and preform well . The odd thing I am seeing is a
> long delay in boot up . Once in the initial loading of the kernel at the "|"
> for 1-2 mins . Then again shortly after printing the kernel banner "freebsd
> 7.3-release etc etc etc" .  This delay is about 1-2 mins as well.  So my
> question does any one know what I could do to speed up the boot up ? I
> suspect the first delay is due to a serial  device probe the second is a
> mystery . also the bios load up time is about 5 mins on this box does anyone
> have any ideas on speeding that up ?
>
> 
> mark saad
> Nonesuch@longcount.org___
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
>

why 7.3? You might have better luck with 8.2. Not I real answer but
sometimes its easier to go around issues
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Re: root on zfs almot working for me but ...

[krad]

On 8 August 2011 09:54, Ollivier Robert  wrote:

> According to Daniel Braniss:
> > Trying to mount root from zfs:z/root
> > ROOT MOUNT ERROR:
> > If you have invalid mount options, reboot, and first try the following
> from
> > the loader prompt:
>
> Have you set the "bootfs" property (see
> http://www.keltia.net/howtos/zfsboot) on the pool?
>
> --
> Ollivier ROBERT -=- FreeBSD: The Power to Serve! -=- robe...@keltia.net
> In memoriam to Ondine, our 2nd child: http://ondine.keltia.net/
>
> ___
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
>

 also have you installed the zfs aware boot blocks?
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Re: root on zfs almost working for me but ...

[krad]

On 8 August 2011 11:30, Daniel Braniss  wrote:

> > On 8/8/11 5:33 AM, Daniel Braniss wrote:
> > >> According to Daniel Braniss:
> > >>> Trying to mount root from zfs:z/root
> > >>> ROOT MOUNT ERROR:
> > >>> If you have invalid mount options, reboot, and first try the
> following from
> > >>> the loader prompt:
> > >>
> > >> Have you set the "bootfs" property (see
> http://www.keltia.net/howtos/zfsboot) on the pool?
> > >>
> > > yes, in my case I did:
> > > zpool set bootfs=z/root z
> > >
> >
> > Have you set zfs_load="YES" in loader.conf and zfs_enable="YES" in
> rc.conf?
> >
> the zfs_load is yes, the rc.conf stuff is for later (i.e init is not yet
> running)
>
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 8.2-STABLE #63 r5807: Sun Aug  7 15:55:02 IDT 2011
>danny@rnd:/home/obj/rnd/r+d/stable/8/sys/HUJI amd64
> Timecounter "i8254" frequency 1193182 Hz quality 0
> CPU: AMD Phenom(tm) II X6 1090T Processor (3214.23-MHz K8-class CPU)
>  Origin = "AuthenticAMD"  Id = 0x100fa0  Family = 10  Model = a  Stepping =
> 0
>
>  Features=0x178bfbff CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
>  Features2=0x802009
>  AMD
> Features=0xee500800 !>
>  AMD
> Features2=0x37ff ,SKINIT,WDT>
>  TSC: P-state invariant
> ...
> ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is
> present;
>to enable, add "vfs.zfs.prefetch_disable=0" to
> /boot/loader.conf.
> ZFS filesystem version 5
> ZFS storage pool version 28
> ...
>
>
> ___
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
>


is the mountpoint for the fs set to legacy?

post your loader.conf here as well zfs the zpool file system hirachy eg zfs
list -r z
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Re: Clustering server in freebsd

[krad]

On 11 October 2011 08:51, Damien Fleuriot  wrote:

>
>
> On 10/10/11 9:31 PM, elman wrote:
> > Dear all
> >
> > I have plan to cluster server with freebsd 8.2 for mailserver. But I'm
> confusing with the software for clustering. Do you have a reference for me,
> or do you have blog and I can see your blog for reference to create
> clustering with freebsd.
> >
> > Thanks hacker
> > Best regards.
> > Mr. L
> > Powered by Telkomsel BlackBerryŽ
> >
>
> Your question is very vague.
>
> - what goal do you want to achieve ?
> - do you want a redundant mail system in case one of your servers goes
> down ?
> - do you want a load balanced system to distribute the load (incurring a
> degraded service if a server goes down) ?
> - what do you mean by "mailserver", is that for outbound and inbound
> email (SMTP), for users to grab their email (POP,IMAP), or both ?
>
>
>
> Basically you're giving us a *means* (clustered servers) to an *end*
> that we do not know/understand.
>
> You've thus had 2 responses so far which might or might not have been
> helpful because nobody knows what you're trying to do ;)
>
> If you want meaningful answers, you'll have to be much more specific.
> ___
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
>

I have built a few resilient mail platform over the past few years, in
highly available, high volume isp environments. In my experience its a bit
more complicated than having a cluster. What you need is a resilient
environment. How much resilience you build in is really down to the business
requirements verses the cost. Some parts of the mail system would make sense
to be clustered, whilst others do not.

SMTP inbound doesnt need clustering resilience takes care of itself by
specifying multiple mx records of the same priority, although a load
balancer does have merit here, but isnt required. Same goes with any spam
filtering you do on the mail. Most mta allow you to specify multiple av/spam
scanners, and round robin their use. Alternatively you can just run them on
localally (not ideal for scalabillity).

SMTP inbound (for pc email clients) you can do dns load balancing but a
proper load balancer is better here.

Pop3, imap, load balanced. You would probably want to make the connections
sticky on relation to the source ip though. You would then get the benefits
of caching etc, by all of the connections from the same user hitting the
same head end. What out from large numbers of users comming from a nated ip
though if you do this.

Webmail head ends - load balanced. Definately need sticky connections here
as session ids get screwed up to easily otherwise.
Webamail back end - Clustered. Most webmail solutions out there require a db
backend (sqwebmail doesnt from what i remember). This has a lot of writes,
therefore needs to be clustered.

Backend authenticaion (webmail/pop3/imap) - depends. If its ldap you can
probably just get away with a master server and multiple slave ldap servers
replicating off this. If its a db (mysql) you could do a similar setup with
reading from some slave servers, or as you most probably have a cluster for
the webmail backend you could use that.

Shared storage: Definitely need to be clustered. Could be a resilient netapp
type thing, sun cluster, red hat cluster, or possibly some kind of hast
solution. Personally I would use a solution based on wafle or zfs. Which you
go for really depends on your budget.

Database clusters: 3 main options here really.
 1. Mysql cluster - could run on freebsd (offical mysql cluster commercial)
you can do a diy cluster though there are a few guides out there.
 2. redhat cluster - supports most dbs
 3. sun cluster -
there are other cluster software available (veritas) but they get very
costly and are probably overkill. WHich ever db solution you go for i'd
recommend you use one that is underpinned with zfs, as it makes backing up
dbs very quick and easy, as well as the data integrity advantages it
provides


Load balancers - there are many commercial options here, although you might
want to look at relayd is you want opensource. Its probably best to run it
on openbsd though as you will get the full range of features then.


As you can see its quite a complicated affair to what seems a simple
question. But it basically boils down to at present there is no proper
clustering solution for freebsd, however there are plenty of roles that
freebsd does an outstanding  and cost efficient job in a resilient
environment
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

possible NFS lockups

[krad]

I have a production mail system with an nfs backend. Every now and again we
see the nfs die on a particular head end. However it doesn't die across all
the nodes. This suggests to me there isnt an issue with the filer itself and
the stats from the filer concur with that.

The symptoms are lines like this appearing in dmesg

nfs server 10.44.17.138:/vol/vol1/mail: not responding
nfs server 10.44.17.138:/vol/vol1/mail: is alive again

trussing df it seems to hang on getfsstat, this is presumably when it tries
the nfs mounts

eg

__sysctl(0xbfbfe224,0x2,0xbfbfe22c,0xbfbfe230,0x0,0x0) = 0 (0x0)
mmap(0x0,1048576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =
1746583552 (0x681ac000)
mmap(0x682ac000,344064,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =
1747632128 (0x682ac000)
munmap(0x681ac000,344064)= 0 (0x0)
getfsstat(0x68201000,0x1270,0x2,0xbfbfe960,0xbfbfe95c,0x1) = 9 (0x9)


I have played with mount options a fair bit but they dont make much
difference. This is what they are set to at present

10.44.17.138:/vol/vol1/mail /mail/0 nfs
rw,noatime,tcp,acdirmax=320,acdirmin=180,acregmax=320,acregmin=180 0   0

When this locking is occuring I find that if I do a show mount or mount
10.44.17.138:/vol/vol1/mail again under another mount point I can access it
fine.

One thing I have just noticed is that lockd and statd always seem to have
died when this happens. Restarting does not help


I find all this a bit perplexing. Can anyone offer any help into why this
might be happening. I have dtrace compliled into the kernel if that could
help with debugging
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Re: possible NFS lockups

[krad]

On 27 July 2010 16:29, krad  wrote:

> I have a production mail system with an nfs backend. Every now and again we
> see the nfs die on a particular head end. However it doesn't die across all
> the nodes. This suggests to me there isnt an issue with the filer itself and
> the stats from the filer concur with that.
>
> The symptoms are lines like this appearing in dmesg
>
> nfs server 10.44.17.138:/vol/vol1/mail: not responding
> nfs server 10.44.17.138:/vol/vol1/mail: is alive again
>
> trussing df it seems to hang on getfsstat, this is presumably when it tries
> the nfs mounts
>
> eg
>
> __sysctl(0xbfbfe224,0x2,0xbfbfe22c,0xbfbfe230,0x0,0x0) = 0 (0x0)
> mmap(0x0,1048576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =
> 1746583552 (0x681ac000)
> mmap(0x682ac000,344064,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =
> 1747632128 (0x682ac000)
> munmap(0x681ac000,344064)= 0 (0x0)
> getfsstat(0x68201000,0x1270,0x2,0xbfbfe960,0xbfbfe95c,0x1) = 9 (0x9)
>
>
> I have played with mount options a fair bit but they dont make much
> difference. This is what they are set to at present
>
> 10.44.17.138:/vol/vol1/mail /mail/0 nfs
> rw,noatime,tcp,acdirmax=320,acdirmin=180,acregmax=320,acregmin=180 0   0
>
> When this locking is occuring I find that if I do a show mount or mount
> 10.44.17.138:/vol/vol1/mail again under another mount point I can access
> it fine.
>
> One thing I have just noticed is that lockd and statd always seem to have
> died when this happens. Restarting does not help
>
>
> I find all this a bit perplexing. Can anyone offer any help into why this
> might be happening. I have dtrace compliled into the kernel if that could
> help with debugging
>

sorry i missed a bit of critical info

# uname -a
FreeBSD X 8.1-STABLE FreeBSD 8.1-STABLE #2: Mon Jul 26 16:10:19 BST 2010
r...@mk-pimap-7.b2b.uk.tiscali.com:/usr/obj/usr/src/sys/DTRACE  i
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Re: pf

[krad]

On 26 September 2010 21:45, jhell  wrote:

> This is more for questions@ or pf@
>
> On 09/26/2010 11:43, Samuel Martín Moro wrote:
> > On Sun, Sep 26, 2010 at 3:34 PM, Michael Powell  >wrote:
> >
> >> Samuel Martín Moro wrote:
> >>
> >>> Hello,
> >>>
> >>>
> >>> I'm trying to set up pf on my soon-to-be new gateway (8.1-RELEASE
> amd64).
> >>> I used the sample configuration file available on
> >>> calomel
> >>> After a few tests, it appears that the gate has fully access to the
> >>> internet, but I can't open connections from clients to distant servers
> >>> (web, ssh, ...).
> >>> Checking pflog log file, I can't see anything about those timeouts,
> even
> >>> if I added the log directive in every block/pass command.
> >>> Everything else seems to work, I can talk with my DNS from the
> internet,
> >>> ssh redirections to another pc also seems to works.
> >>> I just can't access the Internet from a client of my network...
> >>>
> >>> For debugging, I commented out the options and the 'block all in/out'
> >>> directives.
> >>>
> >>> Here's my config file http://pastebin.com/Nim2zBCx
> >>>
> >>> Is there someone understanding what I'm doing wrong?
> >>>
> >> The firewall ruleset is a trifle overly complex for a quick glance;
> study
> >> and analysis would take some doing. However, if you can reach the
> internet
> >> from the firewall box and other client computers behind your NAT can't
> >> (which is what it sounds like you're describing) it may be just that you
> >> are
> >> missing gateway_enable="YES" in your /etc/rc.conf.
> >>
> >> Turning this "ON" makes your firewall box into a router. The status of
> this
> >> can be checked with: sysctl net.inet.ip.forwarding  - a "0" means no
> >> gateway
> >> and a "1" means gateway.
> >>
> >> -Mike
> >>
> >>
> >>
> >> ___
> >> freebsd-questi...@freebsd.org mailing list
> >> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> >> To unsubscribe, send any mail to "
> >> freebsd-questions-unsubscr...@freebsd.org"
> >>
> >
> > the gateway is already enabled (and forwarding is correctly set)
> > whatever, I had to do quick, I started again
> > I think the missing thing on my old conf was the 'scrub' (at least)
> > I made a more simple configuration, as following:
> >
> > ext_if="bge0"
> > int_if="bge1"
> > localnet = $int_if:network
> > emma="10.242.42.200"
> > alpha="10.42.42.42"
> > delta="10.42.42.44"
> > set skip on lo0
> > scrub in on $ext_if all fragment reassemble
> > #INTERNETZ
> > nat on $ext_if from $localnet to any -> ($ext_if)
> > #EMMA
> > rdr on $ext_if inet proto tcp from any to ($ext_if) port 1101 ->
> > $emma port 22
> > rdr on $ext_if inet proto tcp from any to ($ext_if) port 307 ->
> > $emma port 80
> > #WHAT.CD
> > rdr on $ext_if inet proto tcp from any to ($ext_if) port 1666 ->
> > $alpha port 1666
> > #REMOTE ADM
> > rdr on $ext_if inet proto tcp from any to ($ext_if) port 1667 ->
> > $delta port 22
> > rdr on $ext_if inet proto tcp from any to ($ext_if) port 1668 ->
> > $alpha port 22
> > pass in log on $ext_if inet proto tcp from any to $ext_if port 22
> > pass in log on $ext_if inet proto tcp from any to $ext_if port 53
> > pass in log on $ext_if inet proto udp from any to $ext_if port 53
> > pass in log on $ext_if inet proto tcp from any to $ext_if port 1664
> > pass in log on $int_if inet proto tcp from any to any
> > pass in log on $int_if inet proto udp from any to any
> > block in log on $ext_if inet proto icmp from any to $ext_if
> >
> > it's basically working
> > i'll stuff it when I'll have time.
> >
> > Samuel Martín Moro
> > {EPITECH.} tek5
>
>
> --
>
>  jhell,v
> ___
> freebsd-questi...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "
> freebsd-questions-unsubscr...@freebsd.org"
>


its worth doing as restart on pf rather than a reload. Ive seen nat rules
not take affect sometimes on reloads
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Re: Slow disk access while rsync - what should I tune?

[krad]

On 1 November 2010 14:11, Daan Vreeken  wrote:

> Hi Cronfy,
>
> On Saturday 30 October 2010 23:48:45 cronfy wrote:
> > Hello.
> >
> > > Every time backup starts server slows down significantly, disk
> > > operations become very slow. It may take up to 10 seconds to stat() a
> > > file that is not in filesystem cache. At the same time, rsync on
> > > remote server does not affect disk load much, server works without
> > > slowdown.
> >
> > Thank you all for the answers.
> ...
> > Also, is it possible to limit disk operations for rm -rf somehow? The
> > only idea I have at the moment is to replace rm -rf with 'find |
> > slow_down_script | xargs rm' (or use similar patch as for rsync)...
>
> Yes there is. You could use the same 'trick' I've added to rsync and limit
> the
> amount of I/O-creating system calls an application creates.
> You could even create a small wrapper library that does this for a specific
> application, without having to recompile or change the application.
>
> You can find a working proof of concept in "slowdown.c" here :
>http://vehosting.nl/pub_diffs/
>
> The library can be compiled with :
>gcc -Wall -fPIC -shared -o slowdown.so slowdown.c
>
> Then start the application you want to I/O-limit with something like :
>(
>export LD_PRELOAD=slowdown.so
>export LD_LIBRARY_PATH=.:${LD_LIBRARY_PATH}
>ls -R /a/random/huge/directory/
>)
>
> (Assuming you start the application from withing the directory
> where "slowdown.so" resides.)
> This should work with rsync, ls and rm "out of the box", without changing
> the
> source of the applications.
>
>
> Regards,
> --
> Daan Vreeken
> VEHosting
> http://VEHosting.nl
> tel: +31-(0)40-7113050 / +31-(0)6-46210825
> KvK nr: 17174380
> ___
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
>


Just a wild punt here, but are you using zfs on both systems? If you are
look at doing incremental zfs sends as an alternative.
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

RE: Request for opinions - gvinum or ccd?

[krad]

Please don't whack gstripe and zfs together. It should work but is ugly and
you might run into issues. Getting out of them will be harder than a pure
zfs solution

ZFS does support striping by default across vdevs

Eg

Zpool create data da1
Zpool add data da2

Would create a striped data set across da1  and da2

Zpool create data mirror da1 da2
Zpool add data mirror da3 da4

This would create a raid 10 across all drives

Zpool create data raidz2 da1 da2 da3 da5
Zpool add data raidz2 da6 da7 da8 da9

Would create a raid 60

If you replace the add keyword with attach, mirroring is performed rather
than striping

Just for fun here is one of the configs off one of our sun x4500 at work,
its opensolaris not freebsd, but it is zfs. One whoping big array of ~ 28 TB

zpool create -O compression=lzjb -O atime=off data raidz2 c3t0d0 c4t0d0
c8t0d0 c10t0d0 c11t0d0 c3t1d0 c4t1d0 c8t1d0 c9t1d0 c10t1d0 c11t1d0 raidz2
c3t2d0 c4t2d0 c8t2d0 c9t2d0 c11t2d0 c3t3d0 c4t3d0 c8t3d0 c9t3d0 c10t3d0
c11t3d0 raidz2 c3t4d0 c4t4d0 c8t4d0 c10t4d0 c11t4d0 c3t5d0 c4t5d0 c8t5d0
c9t5d0 c10t5d0 c11t5d0 raidz2 c3t6d0 c4t6d0 c8t6d0 c9t6d0 c10t6d0 c11t6d0
c3t7d0 c4t7d0 c9t7d0 c10t7d0 c11t7d0 spare c10t2d0 c8t7d0

$ zpool status
  pool: archive-2
 state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
pool will no longer be accessible on older software versions.
 scrub: scrub completed after 11h9m with 0 errors on Sun May 31 01:09:22
2009
config:

NAME STATE READ WRITE CKSUM
archive-2ONLINE   0 0 0
  raidz2 ONLINE   0 0 0
c3t0d0   ONLINE   0 0 0
c4t0d0   ONLINE   0 0 0
c8t0d0   ONLINE   0 0 0
c10t0d0  ONLINE   0 0 0
c11t0d0  ONLINE   0 0 0
c3t1d0   ONLINE   0 0 0
c4t1d0   ONLINE   0 0 0
c8t1d0   ONLINE   0 0 0
c9t1d0   ONLINE   0 0 0
c10t1d0  ONLINE   0 0 0
c11t1d0  ONLINE   0 0 0
  raidz2 ONLINE   0 0 0
c3t2d0   ONLINE   0 0 0
c4t2d0   ONLINE   0 0 0
c8t2d0   ONLINE   0 0 0
c9t2d0   ONLINE   0 0 0
c11t2d0  ONLINE   0 0 0
c3t3d0   ONLINE   0 0 0
c4t3d0   ONLINE   0 0 0
c8t3d0   ONLINE   0 0 0
c9t3d0   ONLINE   0 0 0
c10t3d0  ONLINE   0 0 0
c11t3d0  ONLINE   0 0 0
  raidz2 ONLINE   0 0 0
c3t4d0   ONLINE   0 0 0
c4t4d0   ONLINE   0 0 0
c8t4d0   ONLINE   0 0 0
c10t4d0  ONLINE   0 0 0
c11t4d0  ONLINE   0 0 0
c3t5d0   ONLINE   0 0 0
c4t5d0   ONLINE   0 0 0
c8t5d0   ONLINE   0 0 0
c9t5d0   ONLINE   0 0 0
c10t5d0  ONLINE   0 0 0
c11t5d0  ONLINE   0 0 0
  raidz2 ONLINE   0 0 0
c3t6d0   ONLINE   0 0 0
c4t6d0   ONLINE   0 0 0
c8t6d0   ONLINE   0 0 0
c9t6d0   ONLINE   0 0 0
c10t6d0  ONLINE   0 0 0
c11t6d0  ONLINE   0 0 0
c3t7d0   ONLINE   0 0 0
c4t7d0   ONLINE   0 0 0
c9t7d0   ONLINE   0 0 0
c10t7d0  ONLINE   0 0 0
c11t7d0  ONLINE   0 0 0
spares
  c10t2d0AVAIL   
  c8t7d0 AVAIL   

errors: No known data errors

ZFS also check sums all data blocks written to the drive so data integrity
is guaranteed. If you are paranoid you can also set it to keep multiple
copies of each file. This will eat up loads of disk space so its best to use
it sparingly one the most important stuff. You can only do it on a fs basis
but this inst a big deal with zfs

Zfs create data/important_stuff
Zfs set copies=3 data/important_stuff

You  can also do compression as well, the big example above has this.

In the near future encryption and deduping are also getting integrated into
zfs. This is probably happening in the next few months on opensolaris, but
if you want those features in freebsd I guess it will take at least 6 months
after that.

With regards to your backup I suggest you definitely look at doing regular
fs snapshots.  To be real safe, id install the tb drive (probably worth
getting another as well 

RE: Request for opinions - gvinum or ccd?

[krad]

Yep it probably isn't clear enough, it does mention stuff about spreading it
across vdevs, but doesn't say striped. But that's sun for you.

The man page should probably be bsdified more as its more or less pulled
from solaris. Note the devices don't look anything like bsd ones (c0t0d0)


-Original Message-
From: Mike Meyer [mailto:m...@mired.org] 
Sent: 31 May 2009 21:33
To: krad
Cc: 'Mike Meyer'; xorquew...@googlemail.com; freebsd-hackers@freebsd.org
Subject: Re: Request for opinions - gvinum or ccd?

On Sun, 31 May 2009 13:13:24 +0100
krad  wrote:

> Please don't whack gstripe and zfs together. It should work but is ugly
and
> you might run into issues. Getting out of them will be harder than a pure
> zfs solution

Yeah, I sorta suspected that might be the case.

> ZFS does support striping by default across vdevs

This isn't documented - at least not in my copies of the manual
page. Not being able to find that was the only reason to even consider
mixing technologies like that.

   Thanks,
http://www.mired.org/consulting.html
Independent Network/Unix/Perforce consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

RE: Request for opinions - gvinum or ccd?

[krad]

Yep, its also worth noting that with the capacities of drives these days you
should really use raidz2 in zfs (or some double parity raid on other
systems) if you are worried about data integrity. The reason being the odds
of the crc checking not detecting an error are much more likely these days.
The extra layer of parity pushes these odds into being much bigger

-Original Message-
From: Wojciech Puchar [mailto:woj...@wojtek.tensor.gdynia.pl] 
Sent: 31 May 2009 22:57
To: xorquew...@googlemail.com
Cc: krad; freebsd-hackers@freebsd.org
Subject: Re: Request for opinions - gvinum or ccd?

>> Would create a striped data set across da1  and da2
>
> What kind of performance gain can I expect from this? I'm purely thinking
> about performance now - the integrity checking stuff of ZFS is a pleasant
> extra.

with stripping - as much as with gstripe, ZFS do roughly the same.

with RAID-z - faster transfer, rougly same IOps as single disk. After i 
read ZFS papers i know that RAID-z is actually more like RAID-3 not RAID-5.




___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

RE: Request for opinions - gvinum or ccd?

[krad]

Zfs has been designed for highly scalable redundant disk pools therefore
using it on a single drive kind of goes against it ethos. Remember a lot of
the blurb in the man page was written by sun and therefore is written with
corporates in mind, therefore the cost with of the data vs an extra drive
being so large why wouldn't you make it redundant.

Having said that sata drives are cheap these days so you would have to be on
the tightest of budgets not to do a mirror.

Having said all this we quite often us zfs on a single drive, well sort of.
The sun clusters have external storage for the shared file systems. These
are usually a bunch of drives, raid 5, 10 or whatever. Then export a single
lun, which is presented to the various nodes. There is a zpool created on
this LUN. So to all intents and purposes zfs thinks its on a single drive
(the redundancy provided by the external array). This is common practice and
we see no issues with it.

-Original Message-
From: owner-freebsd-hack...@freebsd.org
[mailto:owner-freebsd-hack...@freebsd.org] On Behalf Of
xorquew...@googlemail.com
Sent: 01 June 2009 01:00
To: freebsd-hackers@freebsd.org
Subject: Re: Request for opinions - gvinum or ccd?

There is one last thing I'd like clarified. From the zpool
manpage:

  In order  to take advantage of these features, a pool must make use of
  some form of redundancy, using either mirrored or raidz  groups.  While
  ZFS  supports running in a non-redundant configuration, where each root
  vdev is simply a disk or file, this is strongly discouraged.  A  single
  case of bit corruption can render some or all of your data unavailable.

Is this supposed to mean:

  "ZFS is more fragile than most. If you don't use redundancy, one
   case of bit corruption will destroy the filesystem"

Or:

  "Hard disks explode often. Use redundancy."

___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

RE: Request for opinions - gvinum or ccd?

[krad]

In the case of zfs yes, but not always. Eg you could have a concatenated
volume. Where you only start writing to the second disk when the 1st is
full.

-Original Message-
From: Wojciech Puchar [mailto:woj...@wojtek.tensor.gdynia.pl] 
Sent: 01 June 2009 01:31
To: krad
Cc: 'Mike Meyer'; freebsd-hackers@freebsd.org; 'Mike Meyer';
xorquew...@googlemail.com
Subject: RE: Request for opinions - gvinum or ccd?

> Yep it probably isn't clear enough, it does mention stuff about spreading
it
> across vdevs, but doesn't say striped.

isn't spreading and stripping actually the same?

___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

RE: Request for opinions - gvinum or ccd?

[krad]

You shouldn't need to alter the copies attribute to recover from disk
failures as the normal raid should take care of that. What the copies is
useful for is when you get undetected write errors on the drive due to crc
collisions or the drive simply being rubbish

Zfs will intelligently assign the copies across multiple drives if possible,
so if you had 3 vdevs and copies set to three, One copy should end up on
each vdev. Note this isn't the same as mirroring, as each vdev could be a
raidz2 group. With copies=1 you would get a 3rd of the file on each. With
copies 3 you would get a full version of each.

This is obviously very costly on drive space though. It is tunable per file
system though so you don't have to enable it for the whole pool just the
bits you want.

-Original Message-
From: owner-freebsd-hack...@freebsd.org
[mailto:owner-freebsd-hack...@freebsd.org] On Behalf Of Wojciech Puchar
Sent: 01 June 2009 01:26
To: Mike Meyer
Cc: freebsd-hackers@freebsd.org; xorquew...@googlemail.com
Subject: Re: Request for opinions - gvinum or ccd?

>Disks, unlike software, sometimes fail. Using redundancy can help

modern SATA drives fail VERY often. about 30% of drives i bought recently 
failed in less than a year.

> both checksum on and copies > 1 on, and the latter isn't the
> default. It's probably better to let zpool provide the redundancy via
> a mirror or raid configuration than to let zfs do it anyway.

ZFS copies are far from what i consider useful.

for example you set copies=2. You write a file, and get 2 copies.

Then one disk with one copy fails, then you put another, do resilver but 
ZFS DOES NOT rebuild second copy.

You need to write a program that will just rewrite all files to make this.
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

RE: Request for opinions - gvinum or ccd?

[krad]

Its all done on write, so if you update the file it will have multiple
copies again

This explains it quite well

http://blogs.sun.com/relling/entry/zfs_copies_and_data_protection

-Original Message-
From: Wojciech Puchar [mailto:woj...@wojtek.tensor.gdynia.pl] 
Sent: 01 June 2009 10:05
To: krad
Cc: 'Mike Meyer'; freebsd-hackers@freebsd.org; xorquew...@googlemail.com
Subject: RE: Request for opinions - gvinum or ccd?

> You shouldn't need to alter the copies attribute to recover from disk
> failures as the normal raid should take care of that. What the copies is

I don't think we understand each other. I say that when i want 2 copies, 
ZFS should rebuild second copy if it's gone and i run resilver.

it does not, what doesn't make sense for me.


___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

RE: Request for opinions - gvinum or ccd?

[krad]

no you would only loose the data for that block. Zfs also checksums meta
data, but by default keeps multiple copies of it so that's fairly resilient.
If you had the copies set to > 1 then you wouldn't loose the block either,
unless you were real unlucky. 

It's just about pushing the odds back further and further. If you are super
paranoid by all means put in 48 drive, group them into 5 x 8 drive raidz2
vdevs, have a bunch of hot spares, and enable copies=5 for blocks and
metadata, then duplicate the system and put the other box on another
continent and zfs send all you updates every 15 mins via a private
deadicated. This will all prove very resilient, but you will get very little
% storage from your drives, and have quite a large bandwidth bill 8)

Oh and don't forget the scrub you disk regularly. BTW that would rebuild any
missing copies as well (eg if you increase the number of copies after data
is stored on the fs)










-Original Message-
From: Tom Evans [mailto:tevans...@googlemail.com] 
Sent: 01 June 2009 13:50
To: krad
Cc: xorquew...@googlemail.com; freebsd-hackers@freebsd.org
Subject: RE: Request for opinions - gvinum or ccd?

On Mon, 2009-06-01 at 09:32 +0100, krad wrote:
> Zfs has been designed for highly scalable redundant disk pools therefore
> using it on a single drive kind of goes against it ethos. Remember a lot
of
> the blurb in the man page was written by sun and therefore is written with
> corporates in mind, therefore the cost with of the data vs an extra drive
> being so large why wouldn't you make it redundant.
> 
> Having said that sata drives are cheap these days so you would have to be
on
> the tightest of budgets not to do a mirror.
> 
> Having said all this we quite often us zfs on a single drive, well sort
of.
> The sun clusters have external storage for the shared file systems. These
> are usually a bunch of drives, raid 5, 10 or whatever. Then export a
single
> lun, which is presented to the various nodes. There is a zpool created on
> this LUN. So to all intents and purposes zfs thinks its on a single drive
> (the redundancy provided by the external array). This is common practice
and
> we see no issues with it.

By doing this surely you lose a lot of the self healing that ZFS offers?
For instance, if the underlying vdev is just a raid5, then a disk
failure combined with an undetected checksum error on a different disk
would lead you to lose all your data. Or am I missing something?

(PS, top posting is bad)

Tom


___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

RE: Request for opinions - gvinum or ccd?

[krad]

" You need to write a program that will just rewrite all files to make
this."

No you don't you just make sure you scrub the pools regularly once a week
for instance. This way you will hopefully see small block error way before
you have a full drive failure.  

At the end of the day if the data is super critical to you then you should
follow best practices and build an array with sufficient amount of
redundancy in for you needs.

If you cant justify getting 4+ drives to build a raidz2 or raid 10 type
scenario then the data on said array cant actually be worth much itself,
therefore I might be annoying but in the whole scale of things not that
costly if you loose it. If it were you would do thing properly in the first
place



-Original Message-
From: Wojciech Puchar [mailto:woj...@wojtek.tensor.gdynia.pl] 
Sent: 01 June 2009 16:50
To: krad
Cc: 'Mike Meyer'; freebsd-hackers@freebsd.org; xorquew...@googlemail.com
Subject: RE: Request for opinions - gvinum or ccd?

> Its all done on write, so if you update the file it will have multiple
> copies again

what is exactly what i said in the beginning.

___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

RE: Request for opinions - gvinum or ccd?

[krad]

>We remade the pool using 3x 8-drive raidz2 vdevs, and performance has
>been great (400 MBytes/s write, almost 3 GBytes/s sequential read, 800
>MBytes/s random read).

Yep that corresponds with what we saw, although we were getting a little
higher write rates with our 46 drive configuration

___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

RE: sysinstall, GJOURNAL and ZFS

[krad]

Hmm I disagree about large fs have large files. We have inherited quite a
few mail servers at work with 1 TB + fs. They had 10 of millions of files.
When we had a failure and had one reboot it was a nightmare, took ages to
fix. Needless to say this is all on a zfs backed nfs filer now thank god

I do take your point though in many cases what you say would be a good idea.


-Original Message-
From: owner-freebsd-hack...@freebsd.org
[mailto:owner-freebsd-hack...@freebsd.org] On Behalf Of Wojciech Puchar
Sent: 09 June 2009 21:10
To: Dan Naumov
Cc: Nick Barkas; freebsd-curr...@freebsd.org; freebsd-hackers@freebsd.org
Subject: Re: sysinstall, GJOURNAL and ZFS

> filesystems/volumes of today, which can easily span 10tb+ in a
> production environment, having to deal with fsck times is a complete
> no-go.
>
just use large block sizes are really small amount of inodes. it's 
unlikely that you will fill such huge FS with mostly small files, so 
larger blocks are not wasting space.

Just DON'T USE defaults.

fsck takes <5 minutes to check 1TB drive with -b 32768 and -i 524288

> One could of course argue that environments where 10tb+ volumes are
> used right now are special cases, where the administrator would know

No it's not special case. Just needs to be handled properly.

___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

RE: dd copy of FreeBSD-7.2 won't boot

[krad]

Personally id stay away from dd. Create the partitions and file systems
manually, and install the boot loader, then rsync the data across. It will
be a lot faster in most cases, as unlike dd you wont be copying unused
space. Something like this should do the job

Rsync -aPH --exclude=/mnt/** / /mnt

I'm assuming you weren't migrating due to a bad disk

-Original Message-
From: owner-freebsd-hack...@freebsd.org
[mailto:owner-freebsd-hack...@freebsd.org] On Behalf Of Jim Flowers
Sent: 23 June 2009 05:55
To: freebsd-hackers@freebsd.org
Subject: dd copy of FreeBSD-7.2 won't boot

I have a remote server that was dd copied from one hard drive to another - 
essentially the same size.  The disk device name (ad4) is the same but the 
geometry for the new drive has a CHS of 969021/16/63

On booting it hangs at:

F1FreeBSD
Boot: F1

I copied the MBR with 'boot0cfg -B -opacket ad4' just to be sure but no joy.

fbsd fdisk reports start 63, with CHS beg: 0/1/1 end: 1023/15/63.

Any help on direction to solve this?

Thanks.

--
Jim Flowers 

___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Re: Fw: Re: ZFS continuously growing

[krad]

2009/9/3 Robert Eckardt 

> On Thu, 3 Sep 2009 09:09:19 +0200, Adrian Penisoara wrote
> > Hi,
> >
> > On Wed, Sep 2, 2009 at 10:22 PM, Robert Eckardt
> >  wrote:
> > > Do I have to be worried?
> > > Is there a memory leak in the current ZFS implementation?
> > > Why is used space growing slower than free space is shrinking?
> > > Is there some garbage collection needed in ZFS?
> > >
> > > Besides, although the backup server has 3 GB RAM I had to tune arc_max
> > > to 150MB to copy the backed-up data from an 2.8TB ZFS (v6) to the
> > > 4.5 TB ZFS (v13) by "zfs send|zfs recv" without kmalloc panic.
> > > (I.e., the defaults algorithm was not sufficient.)
> >
> >  Do I take you are using ZFS snapshots in between rsync'ing
> > (send/recv requires snapshots) ? Could you please post the "zfs
> > list" output after subsequent runs to clarify ?
> >
> > Regards,
> > Adrian
> > EnterpriseBSD
>
> Hi Adrian,
>
> no I'm not using snapshots. Just seperate directories, where identical
> files are hardlinked by rsync to the version one day older.
> The send|recv was neccessary when I increased the raidz of the backup-fs.
> (Copying everthing to two 1.5TB HDDs and after adding disks back again.
> I used s.th. like "zfs send bigpool/b...@backup | zfs recv big/big".)
>
> Here the zfs list of the last five days:
> Thu Sep  3 09:36:12 CEST 2009  (Today add. 2GB of data were transfered.)
> big  1861882752  0 1861882752 0%5 14545959
> 0%   /big
> big/big  4676727168 2814844416 186188275260% 43137409 14545959
> 75%   /big/big
> NAME  USED  AVAIL  REFER  MOUNTPOINT
> big  2.72T  1.73T  31.5K  /big
> big/big  2.72T  1.73T  2.62T  /big/big
>
> Wed Sep  2 09:36:24 CEST 2009
> big  1869058944128 1869058816 0%5 14602022
> 0%   /big
> big/big  4679698688 2810639872 186905881660% 43226966 14602022
> 75%   /big/big
> NAME  USED  AVAIL  REFER  MOUNTPOINT
> big  2.72T  1.74T  31.5K  /big
> big/big  2.72T  1.74T  2.62T  /big/big
>
> Tue Sep  1 09:36:33 CEST 2009
> big  1875352064  0 1875352064 0%5 14651188
> 0%   /big
> big/big  4683241856 2807889792 187535206460% 43316454 14651188
> 75%   /big/big
> NAME  USED  AVAIL  REFER  MOUNTPOINT
> big  2.71T  1.75T  31.5K  /big
> big/big  2.71T  1.75T  2.62T  /big/big
>
> Mon Aug 31 09:45:26 CEST 2009
> big  1881967616128 1881967488 0%5 14702871
> 0%   /big
> big/big  4686380928 2804413440 188196748860% 43406044 14702871
> 75%   /big/big
> NAME  USED  AVAIL  REFER  MOUNTPOINT
> big  2.71T  1.75T  31.5K  /big
> big/big  2.71T  1.75T  2.61T  /big/big
>
> Sun Aug 30 09:39:31 CEST 2009
> big  1891064192  0 1891064192 0%5 14773939
> 0%   /big
> big/big  4694821376 2803757184 189106419260% 43496712 14773939
> 75%   /big/big
> NAME  USED  AVAIL  REFER  MOUNTPOINT
> big  2.70T  1.76T  31.5K  /big
> big/big  2.70T  1.76T  2.61T  /big/big
>
> Regards,
> Robert
>
> --
> Dr. Robert Eckardt---robert.ecka...@robert-eckardt.de
> ___
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
>

do a " zfs list -t all"

you will see all snapshots and zvols then as well
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Re: Fw: Re: ZFS continuously growing

[krad]

2009/9/3 krad 

>
>
> 2009/9/3 Robert Eckardt 
>
> On Thu, 3 Sep 2009 09:09:19 +0200, Adrian Penisoara wrote
>> > Hi,
>> >
>> > On Wed, Sep 2, 2009 at 10:22 PM, Robert Eckardt
>> >  wrote:
>> > > Do I have to be worried?
>> > > Is there a memory leak in the current ZFS implementation?
>> > > Why is used space growing slower than free space is shrinking?
>> > > Is there some garbage collection needed in ZFS?
>> > >
>> > > Besides, although the backup server has 3 GB RAM I had to tune arc_max
>> > > to 150MB to copy the backed-up data from an 2.8TB ZFS (v6) to the
>> > > 4.5 TB ZFS (v13) by "zfs send|zfs recv" without kmalloc panic.
>> > > (I.e., the defaults algorithm was not sufficient.)
>> >
>> >  Do I take you are using ZFS snapshots in between rsync'ing
>> > (send/recv requires snapshots) ? Could you please post the "zfs
>> > list" output after subsequent runs to clarify ?
>> >
>> > Regards,
>> > Adrian
>> > EnterpriseBSD
>>
>> Hi Adrian,
>>
>> no I'm not using snapshots. Just seperate directories, where identical
>> files are hardlinked by rsync to the version one day older.
>> The send|recv was neccessary when I increased the raidz of the backup-fs.
>> (Copying everthing to two 1.5TB HDDs and after adding disks back again.
>> I used s.th. like "zfs send bigpool/b...@backup | zfs recv big/big".)
>>
>> Here the zfs list of the last five days:
>> Thu Sep  3 09:36:12 CEST 2009  (Today add. 2GB of data were transfered.)
>> big  1861882752  0 1861882752 0%5 14545959
>> 0%   /big
>> big/big  4676727168 2814844416 186188275260% 43137409 14545959
>> 75%   /big/big
>> NAME  USED  AVAIL  REFER  MOUNTPOINT
>> big  2.72T  1.73T  31.5K  /big
>> big/big  2.72T  1.73T  2.62T  /big/big
>>
>> Wed Sep  2 09:36:24 CEST 2009
>> big  1869058944128 1869058816 0%5 14602022
>> 0%   /big
>> big/big  4679698688 2810639872 186905881660% 43226966 14602022
>> 75%   /big/big
>> NAME  USED  AVAIL  REFER  MOUNTPOINT
>> big  2.72T  1.74T  31.5K  /big
>> big/big  2.72T  1.74T  2.62T  /big/big
>>
>> Tue Sep  1 09:36:33 CEST 2009
>> big  1875352064  0 1875352064 0%5 14651188
>> 0%   /big
>> big/big  4683241856 2807889792 187535206460% 43316454 14651188
>> 75%   /big/big
>> NAME  USED  AVAIL  REFER  MOUNTPOINT
>> big  2.71T  1.75T  31.5K  /big
>> big/big  2.71T  1.75T  2.62T  /big/big
>>
>> Mon Aug 31 09:45:26 CEST 2009
>> big  1881967616128 1881967488 0%5 14702871
>> 0%   /big
>> big/big  4686380928 2804413440 188196748860% 43406044 14702871
>> 75%   /big/big
>> NAME  USED  AVAIL  REFER  MOUNTPOINT
>> big  2.71T  1.75T  31.5K  /big
>> big/big  2.71T  1.75T  2.61T  /big/big
>>
>> Sun Aug 30 09:39:31 CEST 2009
>> big  1891064192  0 1891064192 0%5 14773939
>> 0%   /big
>> big/big  4694821376 2803757184 189106419260% 43496712 14773939
>> 75%   /big/big
>> NAME  USED  AVAIL  REFER  MOUNTPOINT
>> big  2.70T  1.76T  31.5K  /big
>> big/big  2.70T  1.76T  2.61T  /big/big
>>
>> Regards,
>> Robert
>>
>> --
>> Dr. Robert Eckardt---robert.ecka...@robert-eckardt.de
>> ___
>> freebsd-hackers@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
>> To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org
>> "
>>
>
> do a " zfs list -t all"
>
> you will see all snapshots and zvols then as well
>
> also might be worth doing a "zfs get all| grep copies"

just to make sure you haven't got multiple copies assigned to any fs

It sounds like for some reason the free blocks are getting put back into the
free block table/map

Also try scrubbing the zpool for good measure. Its worth croning it once a
week
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Re: Fw: Re: ZFS continuously growing [SOLVED]

[krad]

2009/9/3 Robert Eckardt 

> On Thu, 3 Sep 2009 10:01:28 +0100, krad wrote
> > 2009/9/3 Robert Eckardt 
> > On Thu, 3 Sep 2009 09:09:19 +0200, Adrian Penisoara wrote
> >
> > > Hi,
> > >
> > > On Wed, Sep 2, 2009 at 10:22 PM, Robert Eckardt
> > >  wrote:
> >
> > > > Do I have to be worried?
> > > > Is there a memory leak in the current ZFS implementation?
> > > > Why is used space growing slower than free space is shrinking?
> > > > Is there some garbage collection needed in ZFS?
> > > >
> > > > Besides, although the backup server has 3 GB RAM I had to tune
> arc_max
> > > > to 150MB to copy the backed-up data from an 2.8TB ZFS (v6) to the
> > > > 4.5 TB ZFS (v13) by "zfs send|zfs recv" without kmalloc panic.
> > > > (I.e., the defaults algorithm was not sufficient.)
> > >
>
> > do a " zfs list -t all"
> >
> > you will see all snapshots and zvols then as well
>
> Uups, sorry for asking.
> Everything o.k. after "zfs destroy big/b...@backup"  :-(
>
> I hope the info on arc_max will stay useful.
>
> Regards,
> Robert
>
> --
> Dr. Robert Eckardt--- robert.ecka...@robert-eckardt.de
>
>
There was a change between zfs v7 and v13. IN 7 when you did a zfs list it
would show snapshots, after 13 it didnt unless you supplied the switch. It
still catches me out as we have a right mix of zfs version at work, so dont
feel to bad 8)
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Re: HAMMER FS port (status ?)

[krad]

2009/9/24 Gonzalo Nemmi 

> On Thursday 24 September 2009 6:28:31 am Alexander Best wrote:
> > i remember a discussion about HAMMER support on one of the
> > mailingslists which sorta ended with the following statement:
> >
> > "let's get zfs running properly before we even think about starting
> > with HAMMER."
> >
> > cheers.
> > alex
> > ___
> > freebsd-hackers@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> > To unsubscribe, send any mail to
> > "freebsd-hackers-unsubscr...@freebsd.org"
>
> Yup .. that was basically it ...
> Here you go:
> http://lists.freebsd.org/pipermail/freebsd-stable/2008-October/045550.html
>
> Regards
> --
> Blessings
> Gonzalo Nemmi
> ___
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
>


If the istallation get sorted out for zfs its argubly the easiest fs I have
ever had to manage. There are lots of options but you dont have to use them
and if you dont it wont hurt the average user. If you do decide to start
using them at a later date, then its dead easy to.

THe main problem with zfs is resources. If it wasn't for this then there
wouldn't be a good reason not to use zfs default fs (i await the flames 8)
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Re: Distributed SSH attack

[krad]

2009/10/3 Jukka Ruohonen 

> On Fri, Oct 02, 2009 at 05:17:59PM -0400, Greg Larkin wrote:
> > You could set up DenyHosts and contribute to the pool of IPs that are
> > attempting SSH logins on the Net:
> > http://denyhosts.sourceforge.net/faq.html#4_0
>
> While I am well aware that a lot of people use DenyHosts or some equivalent
> tool, I've always been somewhat skeptical about these tools. Few issues:
>
> 1. Firewalls should generally be as static as is possible. There is a
> reason
>   why high securelevel prevents modifications to firewalls.
>
> 2. Generally you do not want some parser to modify your firewall rules.
>   Parsing log entries created by remote unauthenticated users as root is
>   never a good idea.
>
> 3. Doing (2) increases the attack surface.
>
> 4. There have been well-documented cases where (3) has opened opportunities
>   for both remote and local DoS.
>
> Two cents, as they say,
>
> Jukka.
> ___
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
>

simplest this to do is disable password auth, and use key based.
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Fwd: zfs root

[krad]

i got no response on questions, anybody able to answer here?

-- Forwarded message --
From: krad 
Date: 2009/10/3
Subject: zfs root
To: questi...@freebsd.org


Hi,

I have a quick question about freebsd on zfs root. I have built a few test
systems all work fine. I have one question though. Does the loader replay
any information when it accesses the pool?

Basically im interested in how or what is done on reboot after the kernel
panic or power loss. Are there any safe gaurds with regard to the zpool
integrity?
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Re: Superpages on amd64 FreeBSD 7.2-STABLE

[krad]

2009/11/26 Linda Messerschmidt 

> We have a squid proxy process with very large memory requirements (10
> - 20 GB) on a machine with 24GB of RAM.
>
> Unfortunately, we have to rotate the logs of this process once per
> day.  When we do, it fork()s and exec()s about 16-20 child processes
> as helpers.  Since it's got this multi-million-entry page table,
> that's a disaster, because it has to copy all those page table entries
> for each child, then throw them out.  This takes a couple of minutes
> of 100% CPU usage, during which time the machine is pretty much
> unresponsive.
>
> Someone on the squid list suggested we try the new superpages feature
> (vm.pmap.pg_ps_enabled) in 7.2.  We did, and after some tuning, we got
> it to work.
>
> Here's some "sysctl vm.pmap" for a similar machine with 16GB of RAM
> that does NOT have this setting enabled:
>
> vm.pmap.pv_entry_count: 2307899
> vm.pmap.pde.promotions: 0
> vm.pmap.pde.p_failures: 0
> vm.pmap.pde.mappings: 0
> vm.pmap.pde.demotions: 0
> vm.pmap.pv_entry_max: 4276871
> vm.pmap.pg_ps_enabled: 0
>
> Now, here is the machine that does have it, just prior to the daily
> rotation mentioned above:
>
> vm.pmap.pv_entry_count: 61361
> vm.pmap.pde.promotions: 23123
> vm.pmap.pde.p_failures: 327946
> vm.pmap.pde.mappings: 1641
> vm.pmap.pde.demotions: 17848
> vm.pmap.pv_entry_max: 7330186
> vm.pmap.pg_ps_enabled: 1
>
> So it obviously this feature makes a huge difference and is a
> brilliant idea. :-)
>
> My (limited) understanding is that one of the primary benefits of this
> feature is to help situations like ours... a page table that's 512x
> smaller can be copied 512x faster.  However, in practice this doesn't
> happen.  It's like fork() breaks up the squid process into 4kb pages
> again.  Here's the same machine's entries just after rotation:
>
> vm.pmap.pv_entry_count: 1908056
> vm.pmap.pde.promotions: 23212
> vm.pmap.pde.p_failures: 413171
> vm.pmap.pde.mappings: 1641
> vm.pmap.pde.demotions: 21470
> vm.pmap.pv_entry_max: 7330186
> vm.pmap.pg_ps_enabled: 1
>
> So some 3,600 superpages spontaneously turned into 1,850,000 4k pages.
>
> Once this happens, squid seems reluctant to use more superpages until
> its restarted.  We get a lot of p_failures and a slow-but-steady
> stream of demotions.  Here's the same machine just now:
>
> vm.pmap.pv_entry_count: 2022786
> vm.pmap.pde.promotions: 25281
> vm.pmap.pde.p_failures: 996027
> vm.pmap.pde.mappings: 1641
> vm.pmap.pde.demotions: 21683
> vm.pmap.pv_entry_max: 7330186
> vm.pmap.pg_ps_enabled: 1
>
> And a few minutes later:
>
> vm.pmap.pv_entry_count: 2021556
> vm.pmap.pde.promotions: 25331
> vm.pmap.pde.p_failures: 1001773
> vm.pmap.pde.mappings: 1641
> vm.pmap.pde.demotions: 21684
> vm.pmap.pv_entry_max: 7330186
> vm.pmap.pg_ps_enabled: 1
>
> (There were *no* p_failures or demotions in the several hours prior to
> rotation.)
>
> This trend continues... the pv_entry_count bounces up and down even
> though memory usage is increasing, so it's like it's trying to recover
> and convert things back (promotions), but it's having a lot of trouble
> (p_failures).
>
> It's not clear to me if this might be a problem with the superpages
> implementation, or if squid does something particularly horrible to
> its memory when it forks to cause this, but I wanted to ask about it
> on the list in case somebody who understands it better might know
> whats going on. :-)
>
> This is on FreeBSD-STABLE 7.2 amd64 r198976M.
>
> Thanks!
> ___
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
>


Im sure you will get a lot of lovely answers to this but best keep things
simple. WHy not just syslog it of to another server and offload all the
compression to that box. You could even back it with zfs nad do on the fly
gzip compression at the file system level, or use syslog-ng to do it. If you
are worried about zfs and bsd use (open)*solaris  or another filesystem with
with inline compression
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Re: Setting "zfs_arc_max" value in FreeBSD 8.

[krad]

2010/1/19 Ivan Voras 

> On 01/19/10 10:19, Sherin George wrote:
>
>> Hello,
>>
>> I am trying to tune ZFS file system by setting "zfs_arc_max" value in
>> FreeBSD 8.
>>
>> In solaris, it is achieved like this
>>
>> =
>> For example, if an application needs 5 GBytes of memory on a system with
>> 36-GBytes of memory, you could set the arc maximum to 30 GBytes,
>> (0x78000 or 32212254720 bytes). Set the zfs:zfs_arc_max parameter in
>> the
>> /etc/system file:
>>
>> set zfs:zfs_arc_max = 0x78000
>>
>> or
>>
>> set zfs:zfs_arc_max = 32212254720
>> =
>>
>> But, I couldn't find /etc/system file in FreeBSD.
>>
>> Could some one please guide me to correctly configure "zfs_arc_max" in
>> FreeBSD 8.
>>
>
> You should probably start here:
>
> http://wiki.freebsd.org/ZFSTuningGuide
>
> and more generally, here:
>
> http://wiki.freebsd.org/ZFS
>
>
>
> ___
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
>

sysctl vfs.zfs.arc_max=

add it to /etc/sysctl.conf to make persistent
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Re: ZFS'inodes' (as reported by 'df -i') running out?

[krad]

On 18 February 2010 12:04, pluknet  wrote:

> On 18 February 2010 14:41, Ivan Voras  wrote:
> > Karl Pielorz wrote:
> >>
> >> Hi All,
> >>
> >> I originally posted this in freebsd-fs - but didn't get a reply... I
> >> have a number of systems (mostly 7.2-S/amd64) running ZFS. Some of these
> >> handle millions of files.
> >>
> >> I've noticed recently, according to "df -i" I'm starting to run out of
> >> inodes on some of them (96% used).
> >>
> >> e.g.
> >>
> >> "
> >> Filesystem iusedifree %iused Mounted on
> >> vol/imap  1726396   69976   96%  /vol/imap
> >> "
> >>
> >>
> >> I know ZFS doesn't have inodes (think they're znodes), and is capable of
> >> handling more files than you can probably sensibly think about on a
> >> filesystem - but is "df -i" just getting confused, or do I need to be
> >> concerned?
> >
> > AFAIK ZFS allocates inodes when needed so df -i reports the previously
> > allocated value. The number of available inodes should automatically
> > grow as you add more files.
>
> Sorta jfyi. That's what I see on Solaris:
> df: operation not applicable for FSType zfs
>
>
> --
> wbr,
> pluknet
> ___
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
>


Just wait until you start using dedup and get magically growing disks with
df 8))

$ dd if=/dev/urandom of=/tmp/test bs=128k count=1
1+0 records in
1+0 records out
131072 bytes (131 kB) copied, 0.00317671 s, 41.3 MB/s

$ zpool list
NAMESIZE  ALLOC   FREECAP  DEDUP  HEALTH  ALTROOT
dedup  1016M  95.5K  1016M 0%  1.00x  ONLINE  -
rpool   148G   103G  45.0G69%  1.00x  ONLINE  -

$ zfs set dedup=on dedup

$ df -h /dedup
FilesystemSize  Used Avail Use% Mounted on
dedup 984M   21K  984M   1% /dedup


 $ seq 1 1000| while read a ; do cp  /tmp/test /dedup/test.$RANDOM; done

$ df -h /dedup/
FilesystemSize  Used Avail Use% Mounted on
dedup 1.1G  116M  984M  11% /dedup
$ zpool list dedup
NAMESIZE  ALLOC   FREECAP  DEDUP  HEALTH  ALTROOT
dedup  1016M   360K  1016M 0%  921.00x  ONLINE  -


Its only available on opensolaris dev at the moment so dont get to excited,
but in a year or so i mat hit freebsd. You will need a beefy machine though
with a ssd backed l2arc
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Re: Distributed SSH attack

[krad]

On 16 April 2010 09:39, David Xu  wrote:

> Jeremy Lea wrote:
>
>> Hi,
>>
>> This is off topic to this list, but I dont want to subscribe to -chat
>> just to post there...  Someone is currently running a distributed SSH
>> attack against one of my boxes - one attempted login for root every
>> minute or so for the last 48 hours.  They wont get anywhere, since the
>> box in question has no root password, and doesn't allow root logins via
>> SSH anyway...
>>
>> But I was wondering if there were any security researchers out there
>> that might be interested in the +-800 IPs I've collected from the
>> botnet?  The resolvable hostnames mostly appear to be in Eastern Europe
>> and South America - I haven't spotted any that might be 'findable' to
>> get the botnet software.
>>
>> I could switch out the machine for a honeypot in a VM or a jail, by
>> moving the host to a new IP, and if you can think of a way of allowing
>> the next login to succeed with any password, then you could try to see
>> what they delivered...  But I don't have a lot of time to help.
>>
>> Regards,
>>  -Jeremy
>>
>>
> Try to change SSH port to something other than default port 22,
> I always did this for my machines, e.g, change them to 13579 :-)
>
> Regards,
> David Xu
> ___
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
>

dont allow password auth, tcp wrap it, and acl it with pf. Probably more
stuff you can do. Think onions
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Re: Distributed SSH attack

[krad]

On 24 April 2010 14:46, jhell  wrote:

> On 04/16/2010 05:18, krad wrote:
> > On 16 April 2010 09:39, David Xu  wrote:
> >
> >> Jeremy Lea wrote:
> >>
> >>> Hi,
> >>>
> >>> This is off topic to this list, but I dont want to subscribe to -chat
> >>> just to post there...  Someone is currently running a distributed SSH
> >>> attack against one of my boxes - one attempted login for root every
> >>> minute or so for the last 48 hours.  They wont get anywhere, since the
> >>> box in question has no root password, and doesn't allow root logins via
> >>> SSH anyway...
> >>>
> >>> But I was wondering if there were any security researchers out there
> >>> that might be interested in the +-800 IPs I've collected from the
> >>> botnet?  The resolvable hostnames mostly appear to be in Eastern Europe
> >>> and South America - I haven't spotted any that might be 'findable' to
> >>> get the botnet software.
> >>>
> >>> I could switch out the machine for a honeypot in a VM or a jail, by
> >>> moving the host to a new IP, and if you can think of a way of allowing
> >>> the next login to succeed with any password, then you could try to see
> >>> what they delivered...  But I don't have a lot of time to help.
> >>>
> >>> Regards,
> >>>  -Jeremy
> >>>
> >>>
> >> Try to change SSH port to something other than default port 22,
> >> I always did this for my machines, e.g, change them to 13579 :-)
> >>
> >> Regards,
> >> David Xu
> >> ___
> >> freebsd-hackers@freebsd.org mailing list
> >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> >> To unsubscribe, send any mail to "
> freebsd-hackers-unsubscr...@freebsd.org"
> >>
> >
> > dont allow password auth, tcp wrap it, and acl it with pf. Probably more
> > stuff you can do. Think onions
>
> Not allowing password auth also means turning off PAM authentication for
> logins with openssh and has the resulting effect utmp not being updated
> among other things. Be sure you want to go this route.
>
> tcpwrap it ? that is unneeded. The moment you start configuring
> hosts.allow your system is going to be sending requests for ident. Its a
> bad idea with all the other options that are available.
>

Not by default it doesnt.

Even then ident wont happen to random hosts only ones you trust as you will
be protected via pf/ipfw/iptables, its just their as a safety net.

I did mention onions I thought.
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"