Re: Need a filesystem with unlimited inodes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Michael David Crawford wrote: Frank Bonnet wrote: It seems ZFS would match his needs , why don't use it ? Does ZFS really work on FreeBSD? It seems like every day someone is posting about ZFS either getting corrupted or panicking their kernel. Mike The only way to be sure is to test it , I've read at 7.2 it has reached stability. If you have enough money another solution is to buy a NetAPP filer ! -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.11 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkowtKkACgkQ6f7UMO5oSsU6nwCggXRXTOvPYwfgkolmcrHWgIBW SLwAn2dvGbGXjZNFMV9LGxytj6qMJ1RT =bam3 -END PGP SIGNATURE- ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Need a filesystem with unlimited inodes
Mike The only way to be sure is to test it , I've read at 7.2 it has reached stability. As long as there is no recovery tool for ZFS it cannot be treated safe. In SUNs theory it just can't fail - which is nonsense unless machines are perfect and you'll never experience hardware problems. Not disk - but main computer (CPU, memory). It's similar to linux reiserfs i used long time ago. Well it had reiserfsck, but it should be called reiserdestroyfs :) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Need a filesystem with unlimited inodes
On Wednesday 10 June 2009 21:06:06 Karl Vogel wrote:
Create 256 folders named 00-ff:
#!/bin/sh
hex='0 1 2 3 4 5 6 7 8 9 a b c d e f'
for x in $hex ; do
for y in $hex ; do
mkdir ${x}${y}
done
done
exit 0
Or use jot(1) instead of two for loops:
for i in `jot -w %02x 256 0`; do mkdir $i; done
To see the output of the jot in a readable format:
jot -w %02x 256 0 | rs 0 16
Jonathan
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Need a filesystem with unlimited inodes
On Wed, Jun 10, 2009 at 10:17:02PM -0700, Michael David Crawford wrote: Frank Bonnet wrote: It seems ZFS would match his needs , why don't use it ? Does ZFS really work on FreeBSD? It seems like every day someone is posting about ZFS either getting corrupted or panicking their kernel. It mostly seems to depend on the platform. I've got an aging Athlon 64 3200+ with 4GB of memory. I've been using ZFS in some capacity since the early 7.0 RC days. When I'd run FreeBSD/i386 there wasn't much I could do to prevent panics. I did all the recommended tuning mentioned on the FreeBSD wiki and I searched these lists and never could get a stable configuration. However, under amd64 I've never had a panic with ZFS. I was testing some pretty weird stuff, too, and ZFS was rock solid for me. I was running ZFS over geli, often with compression, taking snapshots every minute for an experimental script. Normal usage or stress-testing load, it never failed me. I'd even run ZFS with file-backed devices on top of ZFS or gjournal (both on top of geli), and no stability problems (though I wouldn't recommend such a setup for a production server). I know my account is purely anecdotal, but I've been a unix admin for a long time and I would trust ZFS with my personal data as well as with any any client's data if ZFS met requirements that couldn't be met with UFS2. I don't use ZFS on my workstation because I simply don't need the features and it's slower than UFS. For now, gjournal does an adequate job. -- Geoff P.S. -- I periodically check back at prgmr.com in hopes of seeing official FreeBSD guest support. I love your pricing, but I tend to avoid Linux if I can. Any thoughts on this? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Need a filesystem with unlimited inodes
On Tue, 9 Jun 2009 03:10:46 am Matthew Seaman wrote:
M Or store your data in a RDBMS rather than in the filesystem.
On Tue, 9 Jun 2009 09:45:48 -0500, Kirk Strauser k...@strauser.com said:
K Hear, hear. I'm hard pressed to imagine why you'd need 100M 1KB files.
DBs are great when you have structured data, but semi-structured text
(like email) makes for a very poor fit. To see why, have a look at
http://www.memoryhole.net/~kyle/databaseemail.html
If you really need to store 100 million smallish chunks of information,
consider using zip. Create 256 folders named 00-ff:
#!/bin/sh
hex='0 1 2 3 4 5 6 7 8 9 a b c d e f'
for x in $hex ; do
for y in $hex ; do
mkdir ${x}${y}
done
done
exit 0
Use the hash of your choice to map the name of each chunk to one of 256
zipfiles under each directory. This gives you 64k zipfiles, and if you
put 1500 or so chunks in each one, you're pretty close to 100 million.
me% cat mkchunks
#!/usr/bin/perl -w
for $chunk (@ARGV) {
$_ = chunk2file($chunk);
$file = $1/$2.zip if m/(..)(..)/;
print $file $chunk\n;
}
exit(0);
sub chunk2file {
my $str = shift;
my ($byte, $sum);
use integer;
$sum = 0;
foreach $byte (unpack(C*, $str)) { # SDBM hash
$sum = $byte + 65587 * $sum;
}
$sum = 0x;# keep lowest 16 bits
no integer;
return sprintf(%4.4x, $sum);
}
me% ./mkchunks freebsd solaris
16/f7.zip freebsd
ca/1f.zip solaris
You'll get a better distribution if you use a hash like Digest::SHA1.
--
Karl Vogel I don't speak for the USAF or my company
People like you are the reason people like me need medication. --bumper sticker
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Need a filesystem with unlimited inodes
On 6/8/09, Kelly Jones kelly.terry.jo...@gmail.com wrote: What UFS-like filesystem has unlimited inodes, but is a drop-in replacement for ext3, and is fairly easy to configure? Thanks to everyone who replied. I'm using 100+ rented cloud servers to do stuff for me and rsync the results back to a server I own. I'm sure I could use a db or zip file somehow, but my goal is to get the data centralized ASAP, since I'm paying per hour for the rented cloud servers. DBs and ZIP files would take more time. Once I have the data on my server, I can take as much time as I want. My problem: I ran out of inodes during the rsync process, and had to re-do some of the work, wasting some of my cloud servers time/money. I ended up settling on zfs. It hung hard once (had to reboot the server), which scares me, but it otherwise behaves very well. With compress=gzip-9, it's even saving me disk space. Yes, I realize cloud computing is cheap (and I'm actually still on a free trial), but I value efficiency. -- We're just a Bunch Of Regular Guys, a collective group that's trying to understand and assimilate technology. We feel that resistance to new ideas and technology is unwise and ultimately futile. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Need a filesystem with unlimited inodes
Could you use several large hard drives each with several partitions that each have one filesystem? With eight drives and eight partitions on each, you would multiply the maximum total number of inodes by 256. Mike -- Michael David Crawford m...@prgmr.com prgmr.com - We Don't Assume You Are Stupid. Xen-Powered Virtual Private Servers: http://prgmr.com/xen ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Need a filesystem with unlimited inodes
Michael David Crawford a écrit : Could you use several large hard drives each with several partitions that each have one filesystem? With eight drives and eight partitions on each, you would multiply the maximum total number of inodes by 256. Mike Hello It seems ZFS would match his needs , why don't use it ? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Need a filesystem with unlimited inodes
Frank Bonnet wrote: It seems ZFS would match his needs , why don't use it ? Does ZFS really work on FreeBSD? It seems like every day someone is posting about ZFS either getting corrupted or panicking their kernel. Mike -- Michael David Crawford m...@prgmr.com prgmr.com - We Don't Assume You Are Stupid. Xen-Powered Virtual Private Servers: http://prgmr.com/xen ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Need a filesystem with unlimited inodes
What UFS-like filesystem has unlimited inodes, but is a drop-in replacement for ext3, and is fairly easy to configure? Is UFS2 no longer considered the best general-use filesystem? at least for be it's the best. High performance, minimal hardware resource usage, perfect recovery from failures. using 862 cylinder groups of 118.88MB, 7608 blks, 60864 inodes. I realize I can use f 512 -b 4096 to get 200M+ inodes, but I'm yes do newfs -i 512 -b 4096 -f 512 -U /dev/disk you may add -O1, for simple reason. It will create UFS1 filesystem, which won't be slower in case of small files (it will be with very large), but one inode takes 128, not 256 bytes, which will make huge difference !!! use -m 0 or -m very little as probably you control it yourself what's put here, and extra fragmentation or mostly-full FS won't exist with really small files. just fsck will take a lot of time, you may try gjournal or simply accept it, as FreeBSD don't crash often :) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Need a filesystem with unlimited inodes
At this point you're sort of out of the general-use category :) You want ZFS. Or rather, you don't want to try and fsck a UFS filesystem with 200M inodes. The three drawbacks I can think of to ZFS are it's hard to boot ZFS is very trendy now, but isn't it allocating space in 4KB chunks? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Need a filesystem with unlimited inodes
Dan Nelson wrote: In the last episode (Jun 08), Kelly Jones said: What UFS-like filesystem has unlimited inodes, but is a drop-in replacement for ext3, and is fairly easy to configure? Is UFS2 no longer considered the best general-use filesystem? Reason I ask: I'm going to create many small (~1K) files on a 100G disk and thus need at least 100M inodes. newfs -i maxes out at ~52M inodes (862 groups * 60864 inodes =~ 52M inodes): # newfs -N -i 1 /dev/da1;: same results as -i 2048 /dev/da1: 102400.0MB (209715200 sectors) block size 16384, fragment size 2048 using 862 cylinder groups of 118.88MB, 7608 blks, 60864 inodes. I realize I can use f 512 -b 4096 to get 200M+ inodes, but I'm willing to experiment w/ a new filesystem, provided it behaves mostly like UFS. Thoughts? At this point you're sort of out of the general-use category :) You want ZFS. Or rather, you don't want to try and fsck a UFS filesystem with 200M inodes. The three drawbacks I can think of to ZFS are it's hard to boot from (although you probably aren't booting from da1), it requires more memory than UFS, and there is no ACL support at the moment (not that many people used the ACL support for UFS). If you're already on an amd64 system with 4GB or more RAM you'll be fine. Or store your data in a RDBMS rather than in the filesystem. Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. Flat 3 7 Priory Courtyard PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW, UK signature.asc Description: OpenPGP digital signature
Re: Need a filesystem with unlimited inodes
Or store your data in a RDBMS rather than in the filesystem. sounds like the best solution. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Need a filesystem with unlimited inodes
On Tuesday 09 June 2009 03:10:46 am Matthew Seaman wrote: Or store your data in a RDBMS rather than in the filesystem. Hear, hear. I'm hard pressed to imagine why you'd need 100M 1KB files. -- Kirk Strauser ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Need a filesystem with unlimited inodes
What UFS-like filesystem has unlimited inodes, but is a drop-in replacement for ext3, and is fairly easy to configure? Is UFS2 no longer considered the best general-use filesystem? Reason I ask: I'm going to create many small (~1K) files on a 100G disk and thus need at least 100M inodes. newfs -i maxes out at ~52M inodes (862 groups * 60864 inodes =~ 52M inodes): # newfs -N -i 1 /dev/da1;: same results as -i 2048 /dev/da1: 102400.0MB (209715200 sectors) block size 16384, fragment size 2048 using 862 cylinder groups of 118.88MB, 7608 blks, 60864 inodes. I realize I can use f 512 -b 4096 to get 200M+ inodes, but I'm willing to experiment w/ a new filesystem, provided it behaves mostly like UFS. Thoughts? -- We're just a Bunch Of Regular Guys, a collective group that's trying to understand and assimilate technology. We feel that resistance to new ideas and technology is unwise and ultimately futile. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Need a filesystem with unlimited inodes
In the last episode (Jun 08), Kelly Jones said: What UFS-like filesystem has unlimited inodes, but is a drop-in replacement for ext3, and is fairly easy to configure? Is UFS2 no longer considered the best general-use filesystem? Reason I ask: I'm going to create many small (~1K) files on a 100G disk and thus need at least 100M inodes. newfs -i maxes out at ~52M inodes (862 groups * 60864 inodes =~ 52M inodes): # newfs -N -i 1 /dev/da1;: same results as -i 2048 /dev/da1: 102400.0MB (209715200 sectors) block size 16384, fragment size 2048 using 862 cylinder groups of 118.88MB, 7608 blks, 60864 inodes. I realize I can use f 512 -b 4096 to get 200M+ inodes, but I'm willing to experiment w/ a new filesystem, provided it behaves mostly like UFS. Thoughts? At this point you're sort of out of the general-use category :) You want ZFS. Or rather, you don't want to try and fsck a UFS filesystem with 200M inodes. The three drawbacks I can think of to ZFS are it's hard to boot from (although you probably aren't booting from da1), it requires more memory than UFS, and there is no ACL support at the moment (not that many people used the ACL support for UFS). If you're already on an amd64 system with 4GB or more RAM you'll be fine. -- Dan Nelson dnel...@allantgroup.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
