Re: dd command: BSD analog of conv=fsync?
In the last episode (Nov 18), Thomas Mueller said: What is the (Free)BSD counterpart of conv=fsync in dd command? Command in question is dd if=GNOME-3.6.0.iso of=/dev/DRIVE bs=8M conv=fsync This is for writing to a USB stick, and of course DRIVE is replaced by the actual device node; also I believe bs=8M, good for Linux, would be bs=8m in FreeBSD. I don't really know if conv=fsync is necessary, but that's what was advised in the GNOME test-drive download page. It isn't. Writing to raw devices in FreeBSD immediately writes to the physical media. No flushing is needed. -- Dan Nelson dnel...@allantgroup.com I was able to dd GNOME-3.6.0.iso to that USB stick, a discontinued Kingston Data Traveler model that was inaccessible to NetBSD until they fixed that USB bug recently. I got CAM SCSI error messages in FreeBSD, couldn't access the USB stick in the normal way, but apparently dd worked. These particular Kingston Data Travelers worked normally with previous builds of FreeBSD. That USB stick proved bootable, so I got a test drive of GNOME 3.6.0. I had a difficult time finding my way around the graphical interface,. When I got to a command prompt, I found first there was no nslookup, and then found there was no man command. I thought these were a standard part of (quasi-)Unix OSes. I didn't really get a good impression. Also, the print/text was very small, a recipe for eyestrain. Tom ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: dd command: BSD analog of conv=fsync?
On 2012-11-19 07:42, Thomas Mueller wrote: In the last episode (Nov 18), Thomas Mueller said: What is the (Free)BSD counterpart of conv=fsync in dd command? Command in question is dd if=GNOME-3.6.0.iso of=/dev/DRIVE bs=8M conv=fsync This is for writing to a USB stick, and of course DRIVE is replaced by the actual device node; also I believe bs=8M, good for Linux, would be bs=8m in FreeBSD. I don't really know if conv=fsync is necessary, but that's what was advised in the GNOME test-drive download page. It isn't. Writing to raw devices in FreeBSD immediately writes to the physical media. No flushing is needed. -- Dan Nelson dnel...@allantgroup.com I was able to dd GNOME-3.6.0.iso to that USB stick, a discontinued Kingston Data Traveler model that was inaccessible to NetBSD until they fixed that USB bug recently. I got CAM SCSI error messages in FreeBSD, couldn't access the USB stick in the normal way, but apparently dd worked. These particular Kingston Data Travelers worked normally with previous builds of FreeBSD. That USB stick proved bootable, so I got a test drive of GNOME 3.6.0. I had a difficult time finding my way around the graphical interface,. When I got to a command prompt, I found first there was no nslookup, and then found there was no man command. I thought these were a standard part of (quasi-)Unix OSes. I didn't really get a good impression. Also, the print/text was very small, a recipe for eyestrain. Tom ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org Some Linux distributions tried doing away with nslookup in favor of dig a while back, most have added it back in though. However it looks like you found something that hasn't put it back in. -- Thanks, Dean E. Weimer http://www.dweimer.net/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
fsync: Linux vs FreeBSD
Someone recently posted on one of the PostgreSQL Blogs concerning fsync on Linux/Windows/Mac OS X, but failed to make any comments on any of the BSDs ... the post has to do with how fsync works on the various OSs, and am curious as to whether or not this is something that also afflicts us: http://rhaas.blogspot.com/2010/10/wal-reliability.html From reading our man page, I see no warnings similar to what the other OSs have, specifically: Mac OS X: For applications that require tighter guarantees about the integrity of their data, Mac OS X provides the F_FULLFSYNC fcntl Linux: If the underlying hard disk has write caching enabled, then the data may not really be on permanent storage when fsync() / fdatasync() return. So, do we hide the fact, or are, in fact, not afflicted by this? Thanks ... Marc G. FournierHub.Org Hosting Solutions S.A. scra...@hub.org http://www.hub.org Yahoo:yscrappySkype: hub.orgICQ:7615664MSN:scra...@hub.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: fsync: Linux vs FreeBSD
On Tue, 26 Oct 2010 15:33:52 -0300 (ADT) Marc G. Fournier scra...@hub.org wrote: Someone recently posted on one of the PostgreSQL Blogs concerning fsync on Linux/Windows/Mac OS X, but failed to make any comments on any of the BSDs ... the post has to do with how fsync works on the various OSs, and am curious as to whether or not this is something that also afflicts us: http://rhaas.blogspot.com/2010/10/wal-reliability.html From reading our man page, I see no warnings similar to what the other OSs have, specifically: Mac OS X: For applications that require tighter guarantees about the integrity of their data, Mac OS X provides the F_FULLFSYNC fcntl Linux: If the underlying hard disk has write caching enabled, then the data may not really be on permanent storage when fsync() / fdatasync() return. So, do we hide the fact, or are, in fact, not afflicted by this? Yes, it is a problem: http://leaf.dragonflybsd.org/mailarchive/kernel/2010-01/msg7.html There also a paragraph on write cache and filesystem integrity in da(4): http://www.freebsd.org/cgi/man.cgi?query=daapropos=0sektion=0manpath=FreeBSD+8.1-RELEASEformat=html -- Bruce Cran ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: fsync: Linux vs FreeBSD
On Oct 26, 2010, at 11:33 AM, Marc G. Fournier wrote: Someone recently posted on one of the PostgreSQL Blogs concerning fsync on Linux/Windows/Mac OS X, but failed to make any comments on any of the BSDs ... the post has to do with how fsync works on the various OSs, and am curious as to whether or not this is something that also afflicts us: http://rhaas.blogspot.com/2010/10/wal-reliability.html From reading our man page, I see no warnings similar to what the other OSs have, specifically: Mac OS X: For applications that require tighter guarantees about the integrity of their data, Mac OS X provides the F_FULLFSYNC fcntl Linux: If the underlying hard disk has write caching enabled, then the data may not really be on permanent storage when fsync() / fdatasync() return. So, do we hide the fact, or are, in fact, not afflicted by this? Whether the data actually gets written and the on-disk cache itself flushed seems to depend on a sysctl called hw.ata.wc for FreeBSD or the dkctl setting in NetBSD; write-caching seems to always default to on because otherwise people scream bloody murder about the factor of ten reduction in write performance with it off. Further, by default (ie, FFSv2 with soft updates), data changes are synced out when you do an fsync(), but metadata changes are done asynchronously-- which is exactly what MacOS X does. In other words, if you have write-caching on, no effort is made to invoke ATA_FLUSHCACHE or SCSI SYNCHRONIZE CACHE to make sure that your disk has actually written the bits to permanent storage. [ ... ] http://www.usenix.org/publications/library/proceedings/usenix2000/general/full_papers/seltzer/seltzer_html/index.html Both journaling and Soft Updates systems ensure the integrity of meta-data operations, but they provide slightly different semantics. The four areas of difference are the durability of meta-data operations such as create and delete, the status of the file system after a reboot and recovery, the guarantees made about the data in files after recovery, and the ability to provide atomicity. The original FFS implemented meta-data operations such as create, delete, and rename synchronously, guaranteeing that when the system call returned, the meta-data changes were persistent. Some FFS variants (e.g., Solaris) made deletes asynchronous and other variants (e.g., SVR4) made create and rename asynchronous. However, on FreeBSD, FFS does guarantee that create, delete, and rename operations are synchronous. FFS-async makes no such guarantees, and furthermore does not guarantee that the resulting file system can be recovered (via fsck) to a consistent state after failure. Thus, instead of being a viable candidate for a production file system, FFS-async provides an upper bound on the performance one can expect to achieve with the FFS derivatives. Soft Updates provides looser guarantees than FFS about when meta-data changes reach disk. Create, delete, and rename operations typically reach disk within 45 seconds of the corresponding system call, but can be delayed up to 90 seconds in certain boundary cases (a newly created file in a hierarchy of newly created directories). Regards, -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: fsync: Linux vs FreeBSD
On 10/26/10 21:17, Chuck Swiger wrote: On Oct 26, 2010, at 11:33 AM, Marc G. Fournier wrote: Someone recently posted on one of the PostgreSQL Blogs concerning fsync on Linux/Windows/Mac OS X, but failed to make any comments on any of the BSDs ... the post has to do with how fsync works on the various OSs, and am curious as to whether or not this is something that also afflicts us: http://rhaas.blogspot.com/2010/10/wal-reliability.html From reading our man page, I see no warnings similar to what the other OSs have, specifically: Mac OS X: For applications that require tighter guarantees about the integrity of their data, Mac OS X provides the F_FULLFSYNC fcntl Linux: If the underlying hard disk has write caching enabled, then the data may not really be on permanent storage when fsync() / fdatasync() return. So, do we hide the fact, or are, in fact, not afflicted by this? Whether the data actually gets written and the on-disk cache itself flushed seems to depend on a sysctl called hw.ata.wc for FreeBSD or the dkctl setting in NetBSD; write-caching seems to always default to on because otherwise people scream bloody murder about the factor of ten reduction in write performance with it off. Further, by default (ie, FFSv2 with soft updates), data changes are synced out when you do an fsync(), but metadata changes are done asynchronously-- which is exactly what MacOS X does. In other words, if you have write-caching on, no effort is made to invoke ATA_FLUSHCACHE or SCSI SYNCHRONIZE CACHE to make sure that your disk has actually written the bits to permanent storage. To clarify: all this is in case write-caching happens on disk drives or on disk controllers. The common way to deploy servers for a long time now is to have a disk controller with RAID capabilities and its own RAM cache which is backed by a battery or a capacitor. This controller in turn switches on-drive write caches off. All of the RAID controllers I've seen have a toggle for this last part (on-drive write caches) and it was always turned off by default (though it doesn't hurt to check). To emulate this with desktop drives, as cswiger said, hw.ata.wc should be turned off, with the expected influence on drive performance. All this is valid for UFS. ZFS on the other hand *should* use BIO_FLUSH where appropriate, so it should be safer with desktop drives. OTOH ZFS is so complex that it's hard to say if an error occurs what has caused it. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: fsync: Linux vs FreeBSD
On Oct 26, 2010, at 11:33 AM, Marc G. Fournier wrote:
Someone recently posted on one of the PostgreSQL Blogs concerning fsync on
Linux/Windows/Mac OS X, but failed to make any comments on any of the BSDs
... the post has to do with how fsync works on the various OSs, and am
curious as to whether or not this is something that also afflicts us:
http://rhaas.blogspot.com/2010/10/wal-reliability.html
From reading our man page, I see no warnings similar to what the other OSs
have, specifically:
Mac OS X: For applications that require tighter guarantees about the
integrity of their data, Mac OS X provides the F_FULLFSYNC fcntl
Linux: If the underlying hard disk has write caching enabled, then the
data may not really be on permanent storage when fsync() /
fdatasync() return.
So, do we hide the fact, or are, in fact, not afflicted by this?
It has -always- been the case with the Berkeley FFS that, by default, not
all operations are sychronous, and that sync()/fsync() just 'schedule'
the flush operation, w/o waiting for it to complete. This is precisely
why the tradiitional emergency shutdown was:
sync newline sync newline halt {or haltsys} newline
There is a mount option that forces all I/O on the device to be synchronous,
but the performance penalty is _massive_.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
fsync: giving up on dirty - gjournal on 7.0-R
Hi, On my FreeBSD 7.0-RELEASE amd64 i get the following in my logs when I was upgrading my ports via portmaster. fsync: giving up on dirty 0xff00014745d0: tag devfs, type VCHR usecount 1, writecount 0, refcount 173 mountedhere 0xff00013b2800 flags () v_object 0xff000143f1a0 ref 0 pages 2057 lock type devfs: EXCL (count 1) by thread 0xff0001302340 (pid 39) dev mirror/gm0s3.journal GEOM_JOURNAL: Cannot suspend file system /usr (error=35). gm0s3.journal has a 2GB journal. Journal and data is on the same disk (mirror) Is this something to worry about? Smart for the disks aren't reporting any errors. this is with short tests and extended tests. Regards David N ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: fsync: giving up on dirty - gjournal on 7.0-R
My understanding is that its nothing to worry about. http://freebsd.monkey.org/freebsd-stable/200609/msg00020.html for example, where [EMAIL PROTECTED] who wrote gjounal says It happens sometimes under load, haven't investigated yet what exactly is happening, but you can ignore it for now, it's harmless, it just means journal switch will be done a bit later. Vince David N wrote: Hi, On my FreeBSD 7.0-RELEASE amd64 i get the following in my logs when I was upgrading my ports via portmaster. fsync: giving up on dirty 0xff00014745d0: tag devfs, type VCHR usecount 1, writecount 0, refcount 173 mountedhere 0xff00013b2800 flags () v_object 0xff000143f1a0 ref 0 pages 2057 lock type devfs: EXCL (count 1) by thread 0xff0001302340 (pid 39) dev mirror/gm0s3.journal GEOM_JOURNAL: Cannot suspend file system /usr (error=35). gm0s3.journal has a 2GB journal. Journal and data is on the same disk (mirror) Is this something to worry about? Smart for the disks aren't reporting any errors. this is with short tests and extended tests. Regards David N ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: fsync: giving up on dirty - gjournal on 7.0-R
2008/8/13 Vincent Hoffman [EMAIL PROTECTED]: My understanding is that its nothing to worry about. http://freebsd.monkey.org/freebsd-stable/200609/msg00020.html for example, where [EMAIL PROTECTED] who wrote gjounal says It happens sometimes under load, haven't investigated yet what exactly is happening, but you can ignore it for now, it's harmless, it just means journal switch will be done a bit later. Vince David N wrote: Hi, On my FreeBSD 7.0-RELEASE amd64 i get the following in my logs when I was upgrading my ports via portmaster. fsync: giving up on dirty 0xff00014745d0: tag devfs, type VCHR usecount 1, writecount 0, refcount 173 mountedhere 0xff00013b2800 flags () v_object 0xff000143f1a0 ref 0 pages 2057 lock type devfs: EXCL (count 1) by thread 0xff0001302340 (pid 39) dev mirror/gm0s3.journal GEOM_JOURNAL: Cannot suspend file system /usr (error=35). gm0s3.journal has a 2GB journal. Journal and data is on the same disk (mirror) Is this something to worry about? Smart for the disks aren't reporting any errors. this is with short tests and extended tests. Regards David N ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] Thank you for the info, that puts my mind to rest. Regards David N ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
fsync...??
fsync: giving up on dirty: 0xc4f81d68: tag devfs, type VCHR, usecount 1201, writecount 0, refcount 25, flags (VV_OBJBUF), lock type devfs: EXCL (count 1) by thread 0xc1f43000 (pid 72230) dev da1s1h Can someone tell me what this means? thanks- -- J.D. Bronson Aurora Health Care // Information Services // Milwaukee, WI USA Office: 414.978.8282 // Email: [EMAIL PROTECTED] // Pager: 414.314.8282 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
