Re: WAPBL implementation
On Sat, Nov 21, 2015 at 8:28 PM, Michael McConville wrote: > I agree that this is very cool. It's probably also worth mentioning that > there's a long discussion on NetBSD's tech-kernel@ right now about a > WABPL-related panic. Not sure whether that's relevant to this diff. If I've found the same discussion then this looks like an overflow of journal file. One recommended way to duplicate this issue is removing "a tree". My test currently rsync /usr/src in 7 parallel processes into /raid/1../raid/7 dirs, calculates sha1, unmount, mount, check sha1 and finally rm whole /raid -- which honestly can be considered as "remove the tree" operation I guess. Let's see if the thing panics for me.
Re: WAPBL implementation
Hi Walter, please scratch that. The email was written based on very short experience with wapbl on a small 10gb partition. Now I've performed testing on full 500gb partition all tests done using checksumming SR RAID1. The tests differ only in mount option. rsync: 12m15s (default) -> 7m40s (softdep) -> 6m35s (async) -> 6m5s (wapbl) rm: 5m35s (default) -> 48s (softdep) -> 14s (wapbl) -> 13s (async) so again, fantastic results. To kind of excuse me, I need to add that my fear was caused by seeing huge numbers of small data written using wapbl. After rsync/find/rm cycle I usually detach SR RAID1 drive and this prints some statistic for me. For wapbl based run it looks: RAID1C write statistics in format len (counter/collisions): 512 (917/917), 8192 (75667/293919), 65536 (10941/0), 32768 (204849/0), 4096 (593479/3174400), 12288 (33279/95801), 17408 (88/88), 64512 (279/279), 20480 (12702/21085), 16384 (19738/43332), 28672 (7026/7636), 24576 (9129/11694), 50176 (7/7), 43520 (1/1), 22016 (2/2), 37888 (5/5), 27136 (2/2), 31744 (5/5), 61952 (5/5), 27648 (4/4), 14848 (4/4), 3072 (8/8), 29184 (6/6), 36352 (2/2), 10240 (3/3), 33280 (3/3), 32256 (4/4), 62976 (3/3), 53760 (8/8), 11776 (3/3), 43008 (5/5), 25088 (3/3), 40448 (6/6), 17920 (4/4), 41472 (3/3), 54784 (2/2), 55296 (3/3), 47104 (3/3), 18432 (12/12), 41984 (5/5), 10752 (2/2), 59904 (3/3), 33792 (1/1), 37376 (2/2), 30208 (1/1), 35328 (1/1), 19456 (6/6), 46080 (2/2), 22528 (6/6), 13312 (6/6), 49152 (5/5), 5120 (3/3), 23552 (5/5), 26112 (3/3), 51200 (7/7), 6144 (6/6), 59392 (4/4), 1024 (1/1), 9728 (2/2), 15872 (5/5), 38912 (1/1), 11264 (4/4), 18944 (29/29), 51712 (27/27), 52224 (5/5), 56320 (6/6), 9216 (4/4), 25600 (5/5), 39936 (5/5), 61440 (5/5), 35840 (3/3), 60928 (4/4), 4608 (2/2), 1536 (3/3), 47616 (3/3), 44032 (2/2), 57856 (3/3), 7680 (2/2), 3584 (2/2), 65024 (2/2), 15360 (2/2), 44544 (2/2), 5632 (4/4), 2560 (2/2), 58880 (3/3), 56832 (1/1), 49664 (2/2), 24064 (2/2), 60416 (4/4), 45056 (3/3), 40960 (3/3), 14336 (3/3), 36864 (2/2), 26624 (1/1), 42496 (4/4), 23040 (4/4), 12800 (4/4), 52736 (2/2), 55808 (2/2), 7168 (7/7), 58368 (6/6), 63488 (2/2), 34816 (1/1), 50688 (2/2), 31232 (3/3), 34304 (2/2), 2048 (1/1), 20992 (1/1), 48128 (3/3), 8704 (1/1), 53248 (3/3), 64000 (2/2), 62464 (3/3), 46592 (3/3), 6656 (2/2), 13824 (2/2), 45568 (2/2), 30720 (5/5), 39424 (2/2), 28160 (3/3), 29696 (1/1), 16896 (2/2), RAID1C read statistics in format len (counter): 512 (62), 32768 (826676), 8192 (28), 12288 (14), 4096 (14), 65536 (18086), Please note that collision, where there are not 0, you need to subtract from the collision number the number of I/Os. This is slight issue in my statistics collecting. From the log you can see that all "unusal" operation shows number as (X/X) which means there was not collision and the write was done in a slow way as read chksum block, write data, write chksum block. FYI: statistics from async run looks: RAID1C write statistics in format len (counter/collisions): 512 (2/2), 8192 (75645/293154), 65536 (1154/0), 32768 (203966/0), 4096 (576745/3047106), 12288 (32958/95767), 16384 (19722/43407), 2048 0 (12696/21230), 28672 (7020/7591), 24576 (9125/11989), RAID1C read statistics in format len (counter): 512 (75), 2048 (4), 32768 (826693), 8192 (32), 12288 (16), 4096 (16), 65536 (18166), so you can see that wapbl do all sorts of unusal data lenght operations, but does not hurt SR RAID1C by excess number of collisions with different I/O. You can see such excess number of collisions in case of 4096 bytes lenght write I/O: 4096 (593479/3174400). So well, wapbl looks fantastic so far. I'm now hammering this on top of SR RAID1C and will let you know about any issues... Thanks! Karel On Sat, Nov 21, 2015 at 12:41 PM, Karel Gardas wrote: > RAID1. So here is my question: is there any possibility to convince > current WAPBL code to write transasction into log in 32k blocks with > 32k alignment? I can of course hack the code if you advice where to > test that, I've just so far not been able to find the magic constant > of commit size or so.
Re: WAPBL implementation
I agree that this is very cool. It's probably also worth mentioning that there's a long discussion on NetBSD's tech-kernel@ right now about a WABPL-related panic. Not sure whether that's relevant to this diff.
Re: WAPBL implementation
Walter,
this is really beautiful at least from the performance point of view.
My benchmarks shows:
rsync: 3m27s -> 1m20s
rm: 2m13s -> 9s
so speed increase is here as reported by various wapbl papers which is
really nice.
Anyway, I'd also like to use that with my implementation of
checksumming for SR RAID1. I'm not that lucky like you and under some
conditions my code really sucks performance wise on write operation.
This is due to design/layout of chksumming but even if I experiment
with different layouts (or caching) it still kind of sucks on write.
The only way where it does not suck is if the data are written in 32k
multiplies and aligned on 32k boundary. This way my implementation is
running optimized way hence for write needing only to override chksum
block and write data. And this is done w/o collision with other I/O so
this way is fast and kind of acceptable in comparison with pure SR
RAID1. So here is my question: is there any possibility to convince
current WAPBL code to write transasction into log in 32k blocks with
32k alignment? I can of course hack the code if you advice where to
test that, I've just so far not been able to find the magic constant
of commit size or so.
Thanks!
Karel
PS: for curious my benchmarking shows those numbers. Benchmarked on 2x
500GB WD Re drives, 512 bytes sector size physical. Using my patch to
speedup CRC32 calculation using "by-four" version of algorithm
(360->970 MB/s speed increase). Comparison of pure SR RAID1 with my
chksumming SR RAID1 implementation on a 500GB fs (whole SR RAID
drive). Rsync is copying /usr/src into /raid (1.3GB of data (it's a
git repo) so bigger than usual CVS checkout), rm just rm /raid/* and
find: find /raid -type f cat \{} >/dev/null \;
rsync: 3m27s -> 12m15s
rm: 2m13s -> 5m35s
find+cat: 1m58s -> 2m10s (read benchmark, does not suck IMHO)
dd 2GB write: 16s -> 23s (shows potential of write 32k aligned, for me
very acceptable result)
dd 2GB read: 16s -> 21s
Re: WAPBL implementation
Hello Walter,
Thanks for this crazy diff. It is big so my first comment would be to
split it to ease reviews. Here's a suggestion:
- At least one diff for the kernel parts that are not WAPBL-only. In
other words the refactoring needed for integrating WAPBL into FFS.
This is the critical part in your work since people generally do not
like regression with their filesystem :) From my point of view the
addition of B_CONTIG/B_METAONLY could be easily split. You could
also produce a diff for the wrapper functions that you're adding...
- A diff adding the WAPBL backend to the kernel. This diff would
depend on the previous. It would be nice if you could keep all its
chunks under "#ifdef WAPBL". Even if such define is not strictly
needed for compiling the kernel it helps when reading the code.
- At least one diff for userland tools support.
Splitting your work would also allow you to give a more precise
descriptions about each subpart of this monstrous work :)
Apart from adding WAPBL to OpenBSD, do you have any idea about how to
start using it? Is there any drawback if I'm sharing a media with
WAPBL enabled on an old OpenBSD installation (not supporting WAPBL)?
Concerning WAPBL itself did you consider porting the wapbl(9) manual
from NetBSD? Is its BUGS section still relevant? What about the
Open issues mentioned by Jörg Sonnenberge in its 2009 BSDCan [0] talk,
are they still open?
Keep the good work!
Martin
Some more comments inline.
[0] https://www.bsdcan.org/2009/schedule/attachments/106_wapbl.pdf
> Index: sys/kern/vfs_bio.c
> ===
> RCS file: /Volumes/CSP/cvs/src/sys/kern/vfs_bio.c,v
> retrieving revision 1.170
> diff -u -r1.170 vfs_bio.c
> --- sys/kern/vfs_bio.c19 Jul 2015 16:21:11 - 1.170
> +++ sys/kern/vfs_bio.c23 Oct 2015 20:48:45 -
> @@ -647,6 +672,20 @@
> {
> int s;
>
> + /* If this is a tape block, write the block now. */
> + if (major(bp->b_dev) < nblkdev &&
> + bdevsw[major(bp->b_dev)].d_type == D_TAPE) {
> + bawrite(bp);
> + return;
> + }
If I'm reading NetBSD's log correctly this has been done separately to
avoid calling reassignbuf(). That would be a good candidate for a
single diff :)
> + if (injournal(bp)) {
> + struct mount *mp = wapbl_vptomp(bp->b_vp);
> +
> + if (bp->b_iodone != mp->mnt_wapbl_op->wo_wapbl_biodone)
> + WAPBL_ADD_BUF(mp, bp);
> + }
What I was suggesting before is to keep such blocks under #ifdef WAPBL.
> @@ -743,12 +775,28 @@
>* Determine which queue the buffer should be on, then put it there.
>*/
>
> + /* If it's locked, don't report an error; try again later */
> + if (ISSET(bp->b_flags, (B_LOCKED|B_ERROR)) == (B_LOCKED|B_ERROR))
> + CLR(bp->b_flags, B_ERROR);
> +
> /* If it's not cacheable, or an error, mark it invalid. */
> if (ISSET(bp->b_flags, (B_NOCACHE|B_ERROR)))
> SET(bp->b_flags, B_INVAL);
>
> if (ISSET(bp->b_flags, B_INVAL)) {
> /*
> + * If using WAPBL
> + */
I don't know what's your sync-with-upstream strategy. But by comparing
your diff with NetBSD's sources I feel that the comment below should be
there instead of this one ;)
> + if (ISSET(bp->b_flags, B_LOCKED)) {
> + if (wapbl_vphaswapbl(bp->b_vp)) {
> + struct mount *mp = wapbl_vptomp(bp->b_vp);
> + KASSERT(bp->b_iodone
> + != mp->mnt_wapbl_op->wo_wapbl_biodone);
> + WAPBL_REMOVE_BUF(mp, bp);
> + }
> + }
> +
> + /*
>* If the buffer is invalid, free it now rather than leaving
>* it in a queue and wasting memory.
>*/
^^^
> @@ -1206,6 +1267,17 @@
> }
> #endif
>
> +void
> +buf_adjcnt(struct buf *bp, long ncount)
> +{
> + KASSERT(ncount <= bp->b_bufsize);
> + long ocount = bp->b_bcount;
> + bp->b_bcount = ncount;
> + if (injournal(bp))
> + WAPBL_RESIZE_BUF(wapbl_vptomp(bp->b_vp), bp, bp->b_bufsize,
> + ocount);
> +}
This is an example of wrapper function that could be easily extended :)
> Index: sys/kern/vfs_biomem.c
> ===
> RCS file: /Volumes/CSP/cvs/src/sys/kern/vfs_biomem.c,v
> retrieving revision 1.34
> diff -u -r1.34 vfs_biomem.c
> --- sys/kern/vfs_biomem.c 19 Jul 2015 21:21:14 - 1.34
> +++ sys/kern/vfs_biomem.c 23 Oct 2015 20:48:45 -
> @@ -89,7 +89,7 @@
> {
> splassert(IPL_BIO);
> SET(bp->b_flags, B_BUSY);
> - if (bp->b_data != NULL) {
> + if (bp->b_data != NULL && !(bp->b_flags & B_LOCKED)) {
Style nit. The I
Re: WAPBL implementation
On Wed, Oct 28, 2015 at 12:50:24PM +0100, Jasper Lievisse Adriaanse wrote: > On Wed, Oct 28, 2015 at 09:06:54AM -0200, Walter Neto wrote: > > Adding WAPBL support for dumpfs(8) > > > > next diffs: > > - tunefs(8) showing log information and setting log size > > - fsck_ffs(8) WAPBL support > > > > ok jasper@ > Correction: > I only pointed out that we should have the option to somehow get information > about the log. I did not OK this or the original WAPBL diff. > Sorry guys for the misunderstanding, I learned the language now. Forgive my inexperience > -- > jasper
Re: WAPBL implementation
On Wed, Oct 28, 2015 at 09:06:54AM -0200, Walter Neto wrote: > Adding WAPBL support for dumpfs(8) > > next diffs: > - tunefs(8) showing log information and setting log size > - fsck_ffs(8) WAPBL support > > ok jasper@ Correction: I only pointed out that we should have the option to somehow get information about the log. I did not OK this or the original WAPBL diff. -- jasper
Re: WAPBL implementation
Adding WAPBL support for dumpfs(8)
next diffs:
- tunefs(8) showing log information and setting log size
- fsck_ffs(8) WAPBL support
ok jasper@
Index: sbin/dumpfs/dumpfs.c
===
RCS file: /Volumes/CSP/cvs/src/sbin/dumpfs/dumpfs.c,v
retrieving revision 1.32
diff -u -r1.32 dumpfs.c
--- sbin/dumpfs/dumpfs.c20 Jan 2015 18:22:21 - 1.32
+++ sbin/dumpfs/dumpfs.c28 Oct 2015 10:40:26 -
@@ -40,14 +40,18 @@
#include /* DEV_BSIZE MAXBSIZE isset */
#include
+#include
+#include
#include
+#include
#include
#include
#include
#include
#include
+#include
#include
#include
#include
@@ -68,12 +72,26 @@
} cgun;
#define acgcgun.cg
-intdumpfs(int, const char *);
-intdumpcg(const char *, int, int);
-intmarshal(const char *);
-intopen_disk(const char *);
-void pbits(void *, int);
-__dead voidusage(void);
+union {
+ struct wapbl_wc_header wh;
+ struct wapbl_wc_null wn;
+ char pad[MAXBSIZE];
+} jbuf;
+#define awhjbuf.wh
+#define awnjbuf.wn
+
+int dojournal = 0;
+
+int dumpfs(int, const char *);
+int dumpcg(const char *, int, int);
+int marshal(const char *);
+int open_disk(const char *);
+voidpbits(void *, int);
+int print_journal(const char *, int);
+const char *wapbl_type_string(unsigned);
+voidprint_journal_header(const char *);
+off_t print_journal_entries(const char *, size_t);
+__dead void usage(void);
int
main(int argc, char *argv[])
@@ -84,8 +102,11 @@
domarshal = eval = 0;
- while ((ch = getopt(argc, argv, "m")) != -1) {
+ while ((ch = getopt(argc, argv, "jm")) != -1) {
switch (ch) {
+ case 'j': /* WAPBL journal */
+ dojournal = 1;
+ break;
case 'm':
domarshal = 1;
break;
@@ -258,6 +279,15 @@
afs.fs_cgrotor, afs.fs_fmod, afs.fs_ronly, afs.fs_clean);
printf("avgfpdir %d\tavgfilesize %d\n",
afs.fs_avgfpdir, afs.fs_avgfilesize);
+ if (dojournal) {
+ printf("wapbl version 0x%x\tlocation %u\tflags 0x%x\n",
+ afs.fs_journal_version, afs.fs_journal_location,
+ afs.fs_journal_flags);
+ printf("wapbl loc0 %llu\tloc1 %llu",
+ afs.fs_journallocs[0], afs.fs_journallocs[1]);
+ printf("\tloc2 %llu\tloc3 %llu\n",
+ afs.fs_journallocs[2], afs.fs_journallocs[3]);
+ }
printf("flags\t");
if (afs.fs_magic == FS_UFS2_MAGIC ||
afs.fs_ffs1_flags & FS_FLAGS_UPDATED)
@@ -270,6 +300,8 @@
printf("unclean ");
if (fsflags & FS_DOSOFTDEP)
printf("soft-updates ");
+ if (fsflags & FS_DOWAPBL)
+ printf("wapbl ");
if (fsflags & FS_FLAGS_UPDATED)
printf("updated ");
#if 0
@@ -281,6 +313,10 @@
printf("fsmnt\t%s\n", afs.fs_fsmnt);
printf("volname\t%s\tswuid\t%ju\n",
afs.fs_volname, (uintmax_t)afs.fs_swuid);
+ if (dojournal) {
+ printf("\n");
+ print_journal(name, fd);
+ }
printf("\ncs[].cs_(nbfree,ndir,nifree,nffree):\n\t");
afs.fs_csp = calloc(1, afs.fs_cssize);
for (i = 0, j = 0; i < afs.fs_cssize; i += afs.fs_bsize, j++) {
@@ -457,9 +493,182 @@
printf("\n");
}
+int
+print_journal(const char *name, int fd)
+{
+ daddr_t off;
+ size_t count, blklen, bno, skip;
+ off_t boff, head, tail, len;
+ uint32_t generation;
+
+ if (afs.fs_journal_version != UFS_WAPBL_VERSION)
+ return 0;
+
+ generation = 0;
+ head = tail = 0;
+
+ switch (afs.fs_journal_location) {
+ case UFS_WAPBL_JOURNALLOC_END_PARTITION:
+ case UFS_WAPBL_JOURNALLOC_IN_FILESYSTEM:
+
+ off= afs.fs_journallocs[0];
+ count = afs.fs_journallocs[1];
+ blklen = afs.fs_journallocs[2];
+
+ for (bno=0; bno= 2 * blklen &&
+ ((head >= tail && (boff < tail || boff >= head)) ||
+ (head < tail && (boff >= head && boff < tail
+ continue;
+
+ printf("journal block %lu offset %lld\n",
+ (unsigned long)bno, (long long) boff);
+
+ if (lseek(fd, (off_t)(off*blklen) + boff, SEEK_SET)
+ == (off_t)-1)
+ return (1);
+ if (read(fd, &jbuf, blklen) != (ssize_t)blklen) {
+ warnx("%s: error reading journal", name);
+ return 1;
+ }
+
+ switch (awh.wc_
Re: WAPBL implementation
Walter Neto wrote: > Hi guys, sorry for the inconvenience. > > Once you only have use git at console, cvs is nightmare :( > > But I am learning :) this indeed looks a lot better. although it's quite the diff to review. some brave soul testing of this might be appreciated.
Re: WAPBL implementation
Cool. If possible, also post the diff over http with an URL or something, I don't know if your or my mailers (and the marc.info email archive) mangled the diff, but it was line-wrapped in a few places so I had to edit it for it to apply nicely. 2015-10-23 21:20 GMT+02:00 Walter Neto : > On Fri, Oct 23, 2015 at 09:06:09PM +0200, Janne Johansson wrote: > > did you miss sys/wapbl.h ? > > > > cc ... -D_KERNEL -MD -MP -c ../../../../kern/vfs_bio.c > > ../../../../kern/vfs_bio.c:59:23: error: sys/wapbl.h: No such file or > > directory > > ../../../../kern/vfs_bio.c: In function 'bio_doread': > > ../../../../kern/vfs_bio.c:358: error: 'struct vnode' has no member named > > 'v_specmountpoint' > > > > So sorry Janne, problems with my repo, I will return with the correction. > -- May the most significant bit of your life be positive.
Re: WAPBL implementation
did you miss sys/wapbl.h ?
cc ... -D_KERNEL -MD -MP -c ../../../../kern/vfs_bio.c
../../../../kern/vfs_bio.c:59:23: error: sys/wapbl.h: No such file or
directory
../../../../kern/vfs_bio.c: In function 'bio_doread':
../../../../kern/vfs_bio.c:358: error: 'struct vnode' has no member named
'v_specmountpoint'
2015-10-23 17:19 GMT+02:00 Walter Neto :
> Like recommended from other developers I started developing WAPBL support
> for
> OpenBSD.
>
> Looking at NetBSD and Bitrig I mage a first funcional patch.
>
> Index: sbin/mount/mntopts.h
> ===
> RCS file: /Volumes/CSP/cvs/src/sbin/mount/mntopts.h,v
> retrieving revision 1.16
> diff -u -r1.16 mntopts.h
> --- sbin/mount/mntopts.h13 Jul 2014 12:01:30 - 1.16
> +++ sbin/mount/mntopts.h23 Oct 2015 15:07:07 -
> @@ -66,6 +66,8 @@
> | MFLAG_OPT }
> #define MOPT_SOFTDEP { "softdep",MNT_SOFTDEP, MFLAG_SET }
>
> +#define MOPT_LOG { "log",MNT_LOG, MFLAG_SET }
> +
> /* Control flags. */
> #define MOPT_FORCE { "force", MNT_FORCE, MFLAG_SET }
> #define MOPT_UPDATE{ "update", MNT_UPDATE, MFLAG_SET }
> Index: sbin/mount/mount.c
> ===
> RCS file: /Volumes/CSP/cvs/src/sbin/mount/mount.c,v
> retrieving revision 1.60
> diff -u -r1.60 mount.c
> --- sbin/mount/mount.c 16 Jan 2015 06:39:59 - 1.60
> +++ sbin/mount/mount.c 23 Oct 2015 15:07:07 -
> @@ -94,6 +94,7 @@
> { MNT_ROOTFS, 1, "root file system", "" },
> { MNT_SYNCHRONOUS, 0, "synchronous", "sync" },
> { MNT_SOFTDEP, 0, "softdep", "softdep"
> },
> + { MNT_LOG, 0, "log", "log" },
> { 0,0, "", "" }
> };
>
> Index: sbin/mount_ffs/mount_ffs.c
> ===
> RCS file: /Volumes/CSP/cvs/src/sbin/mount_ffs/mount_ffs.c,v
> retrieving revision 1.21
> diff -u -r1.21 mount_ffs.c
> --- sbin/mount_ffs/mount_ffs.c 16 Jan 2015 06:39:59 - 1.21
> +++ sbin/mount_ffs/mount_ffs.c 23 Oct 2015 15:07:07 -
> @@ -53,6 +53,7 @@
> MOPT_RELOAD,
> MOPT_FORCE,
> MOPT_SOFTDEP,
> + MOPT_LOG,
> { NULL }
> };
>
> Index: sys/conf/GENERIC
> ===
> RCS file: /Volumes/CSP/cvs/src/sys/conf/GENERIC,v
> retrieving revision 1.220
> diff -u -r1.220 GENERIC
> --- sys/conf/GENERIC10 Aug 2015 20:35:36 - 1.220
> +++ sys/conf/GENERIC23 Oct 2015 15:07:07 -
> @@ -43,6 +43,7 @@
> option FIFO# FIFOs; RECOMMENDED
> option TMPFS # efficient memory file system
> option FUSE# FUSE
> +option WAPBL # Write Ahead Physical Block Logging
>
> option SOCKET_SPLICE # Socket Splicing for TCP and UDP
> option TCP_SACK# Selective Acknowledgements for TCP
> Index: sys/conf/files
> ===
> RCS file: /Volumes/CSP/cvs/src/sys/conf/files,v
> retrieving revision 1.604
> diff -u -r1.604 files
> --- sys/conf/files 9 Oct 2015 01:17:21 - 1.604
> +++ sys/conf/files 23 Oct 2015 15:07:07 -
> @@ -732,6 +732,7 @@
> file kern/vfs_vops.c
> file kern/vfs_vnops.c
> file kern/vfs_getcwd.c
> +file kern/vfs_wapbl.c wapbl
> file kern/spec_vnops.c
> file miscfs/deadfs/dead_vnops.c
> file miscfs/fifofs/fifo_vnops.cfifo
> @@ -887,6 +888,7 @@
> file ufs/ffs/ffs_vfsops.c ffs | mfs
> file ufs/ffs/ffs_vnops.c ffs | mfs
> file ufs/ffs/ffs_softdep.c ffs_softupdates
> +file ufs/ffs/ffs_wapbl.c ffs & wapbl
> file ufs/mfs/mfs_vfsops.c mfs
> file ufs/mfs/mfs_vnops.c mfs
> file ufs/ufs/ufs_bmap.cffs | mfs | ext2fs
> @@ -898,6 +900,7 @@
> file ufs/ufs/ufs_quota_stub.c ffs | mfs
> file ufs/ufs/ufs_vfsops.c ffs | mfs | ext2fs
> file ufs/ufs/ufs_vnops.c ffs | mfs | ext2fs
> +file ufs/ufs/ufs_wapbl.c ffs & wapbl
> file ufs/ext2fs/ext2fs_alloc.c ext2fs
> file ufs/ext2fs/ext2fs_balloc.cext2fs
> file ufs/ext2fs/ext2fs_bmap.c ext2fs
> Index: sys/kern/spec_vnops.c
> ===
> RCS file: /Volumes/CSP/cvs/src/sys/kern/spec_vnops.c,v
> retrieving revision 1.83
> diff -u -r1.83 spec_vnops.c
> --- sys/kern/spec_vnops.c 10 Feb 2015 21:56:09 - 1.83
> +++ sys/kern/spec_vnops.c 23 Oct 2015 15:07:07 -
> @@ -408,6 +408,10 @@
> return (EOPNOTSUPP);
> }
>
> +#ifdef WAPBL
> +extern int ffs_wapbl_fsync_vfs(struct v
Re: WAPBL implementation
Pablo Méndez Hernández wrote: > Hi, > > I would say that some information about the authors is missing... I would assume that would be included at the top of the new files, but that does point out they're missing. cvs add, then diff. (with -N if they still don't show up.)
Re: WAPBL implementation
Hi,
I would say that some information about the authors is missing...
Regards.
Pablo
El 23/10/2015 5:25 p. m., "Walter Neto" escribió:
> Like recommended from other developers I started developing WAPBL support
> for
> OpenBSD.
>
> Looking at NetBSD and Bitrig I mage a first funcional patch.
>
> Index: sbin/mount/mntopts.h
> ===
> RCS file: /Volumes/CSP/cvs/src/sbin/mount/mntopts.h,v
> retrieving revision 1.16
> diff -u -r1.16 mntopts.h
> --- sbin/mount/mntopts.h13 Jul 2014 12:01:30 - 1.16
> +++ sbin/mount/mntopts.h23 Oct 2015 15:07:07 -
> @@ -66,6 +66,8 @@
> | MFLAG_OPT }
> #define MOPT_SOFTDEP { "softdep",MNT_SOFTDEP, MFLAG_SET }
>
> +#define MOPT_LOG { "log",MNT_LOG, MFLAG_SET }
> +
> /* Control flags. */
> #define MOPT_FORCE { "force", MNT_FORCE, MFLAG_SET }
> #define MOPT_UPDATE{ "update", MNT_UPDATE, MFLAG_SET }
> Index: sbin/mount/mount.c
> ===
> RCS file: /Volumes/CSP/cvs/src/sbin/mount/mount.c,v
> retrieving revision 1.60
> diff -u -r1.60 mount.c
> --- sbin/mount/mount.c 16 Jan 2015 06:39:59 - 1.60
> +++ sbin/mount/mount.c 23 Oct 2015 15:07:07 -
> @@ -94,6 +94,7 @@
> { MNT_ROOTFS, 1, "root file system", "" },
> { MNT_SYNCHRONOUS, 0, "synchronous", "sync" },
> { MNT_SOFTDEP, 0, "softdep", "softdep"
> },
> + { MNT_LOG, 0, "log", "log" },
> { 0,0, "", "" }
> };
>
> Index: sbin/mount_ffs/mount_ffs.c
> ===
> RCS file: /Volumes/CSP/cvs/src/sbin/mount_ffs/mount_ffs.c,v
> retrieving revision 1.21
> diff -u -r1.21 mount_ffs.c
> --- sbin/mount_ffs/mount_ffs.c 16 Jan 2015 06:39:59 - 1.21
> +++ sbin/mount_ffs/mount_ffs.c 23 Oct 2015 15:07:07 -
> @@ -53,6 +53,7 @@
> MOPT_RELOAD,
> MOPT_FORCE,
> MOPT_SOFTDEP,
> + MOPT_LOG,
> { NULL }
> };
>
> Index: sys/conf/GENERIC
> ===
> RCS file: /Volumes/CSP/cvs/src/sys/conf/GENERIC,v
> retrieving revision 1.220
> diff -u -r1.220 GENERIC
> --- sys/conf/GENERIC10 Aug 2015 20:35:36 - 1.220
> +++ sys/conf/GENERIC23 Oct 2015 15:07:07 -
> @@ -43,6 +43,7 @@
> option FIFO# FIFOs; RECOMMENDED
> option TMPFS # efficient memory file system
> option FUSE# FUSE
> +option WAPBL # Write Ahead Physical Block Logging
>
> option SOCKET_SPLICE # Socket Splicing for TCP and UDP
> option TCP_SACK# Selective Acknowledgements for TCP
> Index: sys/conf/files
> ===
> RCS file: /Volumes/CSP/cvs/src/sys/conf/files,v
> retrieving revision 1.604
> diff -u -r1.604 files
> --- sys/conf/files 9 Oct 2015 01:17:21 - 1.604
> +++ sys/conf/files 23 Oct 2015 15:07:07 -
> @@ -732,6 +732,7 @@
> file kern/vfs_vops.c
> file kern/vfs_vnops.c
> file kern/vfs_getcwd.c
> +file kern/vfs_wapbl.c wapbl
> file kern/spec_vnops.c
> file miscfs/deadfs/dead_vnops.c
> file miscfs/fifofs/fifo_vnops.cfifo
> @@ -887,6 +888,7 @@
> file ufs/ffs/ffs_vfsops.c ffs | mfs
> file ufs/ffs/ffs_vnops.c ffs | mfs
> file ufs/ffs/ffs_softdep.c ffs_softupdates
> +file ufs/ffs/ffs_wapbl.c ffs & wapbl
> file ufs/mfs/mfs_vfsops.c mfs
> file ufs/mfs/mfs_vnops.c mfs
> file ufs/ufs/ufs_bmap.cffs | mfs | ext2fs
> @@ -898,6 +900,7 @@
> file ufs/ufs/ufs_quota_stub.c ffs | mfs
> file ufs/ufs/ufs_vfsops.c ffs | mfs | ext2fs
> file ufs/ufs/ufs_vnops.c ffs | mfs | ext2fs
> +file ufs/ufs/ufs_wapbl.c ffs & wapbl
> file ufs/ext2fs/ext2fs_alloc.c ext2fs
> file ufs/ext2fs/ext2fs_balloc.cext2fs
> file ufs/ext2fs/ext2fs_bmap.c ext2fs
> Index: sys/kern/spec_vnops.c
> ===
> RCS file: /Volumes/CSP/cvs/src/sys/kern/spec_vnops.c,v
> retrieving revision 1.83
> diff -u -r1.83 spec_vnops.c
> --- sys/kern/spec_vnops.c 10 Feb 2015 21:56:09 - 1.83
> +++ sys/kern/spec_vnops.c 23 Oct 2015 15:07:07 -
> @@ -408,6 +408,10 @@
> return (EOPNOTSUPP);
> }
>
> +#ifdef WAPBL
> +extern int ffs_wapbl_fsync_vfs(struct vnode *, int);
> +#endif
> +
> /*
> * Synch buffers associated with a block device
> */
> @@ -422,6 +426,15 @@
>
> if (vp->v_type == VCHR)
> return (0);
> +
> +
> +#ifdef WAPBL
> + i
Re: WAPBL implementation
On Fri, Oct 23, 2015 at 01:19:15PM -0200, Walter Neto wrote:
> Like recommended from other developers I started developing WAPBL support for
> OpenBSD.
>
> Looking at NetBSD and Bitrig I mage a first funcional patch.
Wow... that is a big diff :)
Care to elaborate in some more words what "functional" means here?
Thanks,
Regards,
Joerg
> Index: sbin/mount/mntopts.h
> ===
> RCS file: /Volumes/CSP/cvs/src/sbin/mount/mntopts.h,v
> retrieving revision 1.16
> diff -u -r1.16 mntopts.h
> --- sbin/mount/mntopts.h 13 Jul 2014 12:01:30 - 1.16
> +++ sbin/mount/mntopts.h 23 Oct 2015 15:07:07 -
> @@ -66,6 +66,8 @@
> | MFLAG_OPT }
> #define MOPT_SOFTDEP { "softdep",MNT_SOFTDEP, MFLAG_SET }
>
> +#define MOPT_LOG { "log",MNT_LOG, MFLAG_SET }
> +
> /* Control flags. */
> #define MOPT_FORCE { "force", MNT_FORCE, MFLAG_SET }
> #define MOPT_UPDATE { "update", MNT_UPDATE, MFLAG_SET }
> Index: sbin/mount/mount.c
> ===
> RCS file: /Volumes/CSP/cvs/src/sbin/mount/mount.c,v
> retrieving revision 1.60
> diff -u -r1.60 mount.c
> --- sbin/mount/mount.c16 Jan 2015 06:39:59 - 1.60
> +++ sbin/mount/mount.c23 Oct 2015 15:07:07 -
> @@ -94,6 +94,7 @@
> { MNT_ROOTFS, 1, "root file system", "" },
> { MNT_SYNCHRONOUS, 0, "synchronous", "sync" },
> { MNT_SOFTDEP, 0, "softdep", "softdep" },
> + { MNT_LOG, 0, "log", "log" },
> { 0,0, "", "" }
> };
>
> Index: sbin/mount_ffs/mount_ffs.c
> ===
> RCS file: /Volumes/CSP/cvs/src/sbin/mount_ffs/mount_ffs.c,v
> retrieving revision 1.21
> diff -u -r1.21 mount_ffs.c
> --- sbin/mount_ffs/mount_ffs.c16 Jan 2015 06:39:59 - 1.21
> +++ sbin/mount_ffs/mount_ffs.c23 Oct 2015 15:07:07 -
> @@ -53,6 +53,7 @@
> MOPT_RELOAD,
> MOPT_FORCE,
> MOPT_SOFTDEP,
> + MOPT_LOG,
> { NULL }
> };
>
> Index: sys/conf/GENERIC
> ===
> RCS file: /Volumes/CSP/cvs/src/sys/conf/GENERIC,v
> retrieving revision 1.220
> diff -u -r1.220 GENERIC
> --- sys/conf/GENERIC 10 Aug 2015 20:35:36 - 1.220
> +++ sys/conf/GENERIC 23 Oct 2015 15:07:07 -
> @@ -43,6 +43,7 @@
> option FIFO# FIFOs; RECOMMENDED
> option TMPFS # efficient memory file system
> option FUSE# FUSE
> +option WAPBL # Write Ahead Physical Block Logging
>
> option SOCKET_SPLICE # Socket Splicing for TCP and UDP
> option TCP_SACK# Selective Acknowledgements for TCP
> Index: sys/conf/files
> ===
> RCS file: /Volumes/CSP/cvs/src/sys/conf/files,v
> retrieving revision 1.604
> diff -u -r1.604 files
> --- sys/conf/files9 Oct 2015 01:17:21 - 1.604
> +++ sys/conf/files23 Oct 2015 15:07:07 -
> @@ -732,6 +732,7 @@
> file kern/vfs_vops.c
> file kern/vfs_vnops.c
> file kern/vfs_getcwd.c
> +file kern/vfs_wapbl.cwapbl
> file kern/spec_vnops.c
> file miscfs/deadfs/dead_vnops.c
> file miscfs/fifofs/fifo_vnops.c fifo
> @@ -887,6 +888,7 @@
> file ufs/ffs/ffs_vfsops.cffs | mfs
> file ufs/ffs/ffs_vnops.c ffs | mfs
> file ufs/ffs/ffs_softdep.c ffs_softupdates
> +file ufs/ffs/ffs_wapbl.c ffs & wapbl
> file ufs/mfs/mfs_vfsops.cmfs
> file ufs/mfs/mfs_vnops.c mfs
> file ufs/ufs/ufs_bmap.c ffs | mfs | ext2fs
> @@ -898,6 +900,7 @@
> file ufs/ufs/ufs_quota_stub.cffs | mfs
> file ufs/ufs/ufs_vfsops.cffs | mfs | ext2fs
> file ufs/ufs/ufs_vnops.c ffs | mfs | ext2fs
> +file ufs/ufs/ufs_wapbl.c ffs & wapbl
> file ufs/ext2fs/ext2fs_alloc.c ext2fs
> file ufs/ext2fs/ext2fs_balloc.c ext2fs
> file ufs/ext2fs/ext2fs_bmap.cext2fs
> Index: sys/kern/spec_vnops.c
> ===
> RCS file: /Volumes/CSP/cvs/src/sys/kern/spec_vnops.c,v
> retrieving revision 1.83
> diff -u -r1.83 spec_vnops.c
> --- sys/kern/spec_vnops.c 10 Feb 2015 21:56:09 - 1.83
> +++ sys/kern/spec_vnops.c 23 Oct 2015 15:07:07 -
> @@ -408,6 +408,10 @@
> return (EOPNOTSUPP);
> }
>
> +#ifdef WAPBL
> +extern int ffs_wapbl_fsync_vfs(struct vnode *, int);
> +#endif
> +
> /*
> * Synch buffers associated with a block device
> */
> @@ -422,6 +426,15 @@
>
> if (vp->v_type == VCHR)
>
WAPBL implementation
Like recommended from other developers I started developing WAPBL support for
OpenBSD.
Looking at NetBSD and Bitrig I mage a first funcional patch.
Index: sbin/mount/mntopts.h
===
RCS file: /Volumes/CSP/cvs/src/sbin/mount/mntopts.h,v
retrieving revision 1.16
diff -u -r1.16 mntopts.h
--- sbin/mount/mntopts.h13 Jul 2014 12:01:30 - 1.16
+++ sbin/mount/mntopts.h23 Oct 2015 15:07:07 -
@@ -66,6 +66,8 @@
| MFLAG_OPT }
#define MOPT_SOFTDEP { "softdep",MNT_SOFTDEP, MFLAG_SET }
+#define MOPT_LOG { "log",MNT_LOG, MFLAG_SET }
+
/* Control flags. */
#define MOPT_FORCE { "force", MNT_FORCE, MFLAG_SET }
#define MOPT_UPDATE{ "update", MNT_UPDATE, MFLAG_SET }
Index: sbin/mount/mount.c
===
RCS file: /Volumes/CSP/cvs/src/sbin/mount/mount.c,v
retrieving revision 1.60
diff -u -r1.60 mount.c
--- sbin/mount/mount.c 16 Jan 2015 06:39:59 - 1.60
+++ sbin/mount/mount.c 23 Oct 2015 15:07:07 -
@@ -94,6 +94,7 @@
{ MNT_ROOTFS, 1, "root file system", "" },
{ MNT_SYNCHRONOUS, 0, "synchronous", "sync" },
{ MNT_SOFTDEP, 0, "softdep", "softdep" },
+ { MNT_LOG, 0, "log", "log" },
{ 0,0, "", "" }
};
Index: sbin/mount_ffs/mount_ffs.c
===
RCS file: /Volumes/CSP/cvs/src/sbin/mount_ffs/mount_ffs.c,v
retrieving revision 1.21
diff -u -r1.21 mount_ffs.c
--- sbin/mount_ffs/mount_ffs.c 16 Jan 2015 06:39:59 - 1.21
+++ sbin/mount_ffs/mount_ffs.c 23 Oct 2015 15:07:07 -
@@ -53,6 +53,7 @@
MOPT_RELOAD,
MOPT_FORCE,
MOPT_SOFTDEP,
+ MOPT_LOG,
{ NULL }
};
Index: sys/conf/GENERIC
===
RCS file: /Volumes/CSP/cvs/src/sys/conf/GENERIC,v
retrieving revision 1.220
diff -u -r1.220 GENERIC
--- sys/conf/GENERIC10 Aug 2015 20:35:36 - 1.220
+++ sys/conf/GENERIC23 Oct 2015 15:07:07 -
@@ -43,6 +43,7 @@
option FIFO# FIFOs; RECOMMENDED
option TMPFS # efficient memory file system
option FUSE# FUSE
+option WAPBL # Write Ahead Physical Block Logging
option SOCKET_SPLICE # Socket Splicing for TCP and UDP
option TCP_SACK# Selective Acknowledgements for TCP
Index: sys/conf/files
===
RCS file: /Volumes/CSP/cvs/src/sys/conf/files,v
retrieving revision 1.604
diff -u -r1.604 files
--- sys/conf/files 9 Oct 2015 01:17:21 - 1.604
+++ sys/conf/files 23 Oct 2015 15:07:07 -
@@ -732,6 +732,7 @@
file kern/vfs_vops.c
file kern/vfs_vnops.c
file kern/vfs_getcwd.c
+file kern/vfs_wapbl.c wapbl
file kern/spec_vnops.c
file miscfs/deadfs/dead_vnops.c
file miscfs/fifofs/fifo_vnops.cfifo
@@ -887,6 +888,7 @@
file ufs/ffs/ffs_vfsops.c ffs | mfs
file ufs/ffs/ffs_vnops.c ffs | mfs
file ufs/ffs/ffs_softdep.c ffs_softupdates
+file ufs/ffs/ffs_wapbl.c ffs & wapbl
file ufs/mfs/mfs_vfsops.c mfs
file ufs/mfs/mfs_vnops.c mfs
file ufs/ufs/ufs_bmap.cffs | mfs | ext2fs
@@ -898,6 +900,7 @@
file ufs/ufs/ufs_quota_stub.c ffs | mfs
file ufs/ufs/ufs_vfsops.c ffs | mfs | ext2fs
file ufs/ufs/ufs_vnops.c ffs | mfs | ext2fs
+file ufs/ufs/ufs_wapbl.c ffs & wapbl
file ufs/ext2fs/ext2fs_alloc.c ext2fs
file ufs/ext2fs/ext2fs_balloc.cext2fs
file ufs/ext2fs/ext2fs_bmap.c ext2fs
Index: sys/kern/spec_vnops.c
===
RCS file: /Volumes/CSP/cvs/src/sys/kern/spec_vnops.c,v
retrieving revision 1.83
diff -u -r1.83 spec_vnops.c
--- sys/kern/spec_vnops.c 10 Feb 2015 21:56:09 - 1.83
+++ sys/kern/spec_vnops.c 23 Oct 2015 15:07:07 -
@@ -408,6 +408,10 @@
return (EOPNOTSUPP);
}
+#ifdef WAPBL
+extern int ffs_wapbl_fsync_vfs(struct vnode *, int);
+#endif
+
/*
* Synch buffers associated with a block device
*/
@@ -422,6 +426,15 @@
if (vp->v_type == VCHR)
return (0);
+
+
+#ifdef WAPBL
+ if (vp->v_type == VBLK &&
+ vp->v_specmountpoint != NULL &&
+ vp->v_specmountpoint->mnt_wapbl != NULL)
+ return (ffs_wapbl_fsync_vfs(vp, ap->a_waitfor));
+#endif
+
/*
* Flush all dirty buffers associated with a block device.
*/
Index: sys/kern/vfs_bio.c
===
