Re: [zfs-discuss] zfs streams data corruption
Miles Nordin wrote: gp Performing a checkpoint will perform such tasks as making sure gp that all transactions recorded in the log but not yet written gp to the database are written out and that the system is not in gp the middle of a write when you grab the data. great copying of buzzwords out of a glossary, Wasn't copied from a glossary, I just tried to simplify it enough for you to understand. I apologize if I didn't accomplish that goal. but does it change my claim or not? My claim is: that SQLite2 should be equally as tolerant of snapshot backups as it is of cord-yanking. You're missing the point here Miles. The folks weren't asking for a method to confirm their database was able to perform proper error recovery and confirm it would survive having the cord yanked out of the wall. They were asking for a reliable way to backup their data. The best way to do that is not by snapshotting alone. The process of performing database backups is well understood and supported throughout the industry. Relying on the equivalent of crashing the database to perform backups isn't how professionals get the job done. There is a reason that database vendor do not suggest you backup their databases by pulling the plug out of the wall or killing the running process. The best way to backup a database is by using a checkpoint. Your comment about checkpoints being for systems where snapshots are not available is not accurate. That is the normal method of backing up databases under Solaris among others. Checkpoints are useful for all systems since they guarantee that the database files are consistent and do not require recovery which doesn't always work no matter what the glossy brochures tell you. Typically they are used in concert with snapshots. Force the checkpoint, trigger the snapshot and you're golden. Let's take a simple case of a transaction which consists of three database updates within a transaction. One of those writes succeeds, you take a snapshot and then the two other writes succeed. Everyone concerned with the transaction believes it succeeded but your snapshot does not show that. When the database starts up again, the data it will have in your snapshot indicates the transaction never succeeded and therefore it will roll out the database transaction and you will lose that transaction. Well, it will assuming that all code involved in that recovery works flawlessly. Issuing a checkpoint on the other hand causes the database to complete the transaction including ensuring consistency of the database files before you take your snapshot. NOTE: If you issue a checkpoint and then perform a snapshot you will get consistent data which does not require the database perform recovery. Matter of fact, that's the best way to do it. Your dismissal of write activity taking place is inaccurate. Snapshots take a picture of the file system at a point in time. They have no knowledge of whether or not one of three writes required for the database to be consistent have completed. (Refer to above example) Data does not hit the disk instantly, it takes some finite amount of time in between when the write command is issued for it to arrive at the disk. Plainly, terminating the writes between when they are issued and before it has completed is possible and a matter of timing. The database on the other hand does understand when the transaction has completed and allows outside processes to take advantage of this knowledge via checkpointing. All real database systems have flaws in the recovery process and so far every database system I've seen has had issues at one time or another. If we were in a perfect world it SHOULD work every time but we aren't in a perfect world. ZFS promises on disk consistency but as we saw in the recent thread about Unreliable for professional usage it is possible to have issues. Likewise with database systems. Regards, Greg ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs streams data corruption
Miles Nordin wrote: Hope this helps untangle some FUD. Snapshot backups of databases *are* safe, unless the database or application above it is broken in a way that makes cord-yanking unsafe too. Actually Miles, what they were asking for is generally referred to as a checkpoint and they are used by all major databases for backing up files. Performing a checkpoint will perform such tasks as making sure that all transactions recorded in the log but not yet written to the database are written out and that the system is not in the middle of a write when you grab the data. Dragging the discussion of database recovery into the discussion seems to me to only be increasing the FUD factor. Regards, Greg ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS: unreliable for professional usage?
Miles Nordin wrote: gm That implies that ZFS will have to detect removable devices gm and treat them differently than fixed devices. please, no more of this garbage, no more hidden unchangeable automatic condescending behavior. The whole format vs rmformat mess is just ridiculous. And software and hardware developers alike have both proven themselves incapable of settling on a definition of ``removeable'' that fits with actual use-cases like: FC/iSCSI; hot-swappable SATA; adapters that have removeable sockets on both ends like USB-to-SD, firewire CD-ROM's, SATA/SAS port multipliers, and so on. Since this discussion is taking place in the context of someone removing a USB stick I think you're confusing the issue by dragging in other technologies. Let's keep this in the context of the posts preceding it which is how USB devices are treated. I would argue that one of the first design goals in an environment where you can expect people who are not computer professionals to be interfacing with computers is to make sure that the appropriate safeties are in place and that the system does not behave in a manner which a reasonable person might find unexpected. This is common practice for any sort of professional engineering effort. As an example, you aren't going to go out there and find yourself a chainsaw being sold new without a guard. It might be removable, but the default is to include it. Why? Well because there is a considerable chance of damage to the user without it. Likewise with a file system on a device which might cache a data write for as long as thirty seconds while being easily removable. In this case, the user may write the file and seconds later remove the device. Many folks out there behave in this manner. It really doesn't matter to them that they have a copy of the last save they did two hours ago, what they want and expect is that the most recent data they saved actually be on the USB stick for the to retrieve. What you are suggesting is that it is better to lose that data when it could have been avoided. I would personally suggest that it is better to have default behavior which is not surprising along with more advanced behavior for those who have bothered to read the manual. In Windows case, the write cache can be turned on, it is not unchangeable and those who have educated themselves use it. I seldom turn it on unless I'm doing heavy I/O to a USB hard drive, otherwise the performance difference is just not that great. Regards, Greg ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS: unreliable for professional usage?
Ross wrote: I can also state with confidence that very, very few of the 100 staff working here will even be aware that it's possible to unmount a USB volume in windows. They will all just pull the plug when their work is saved, and since they all come to me when they have problems, I think I can safely say that pulling USB devices really doesn't tend to corrupt filesystems in Windows. Everybody I know just waits for the light on the device to go out. The key here is that Windows does not cache writes to the USB drive unless you go in and specifically enable them. It caches reads but not writes. If you enable them you will lose data if you pull the stick out before all the data is written. This is the type of safety measure that needs to be implemented in ZFS if it is to support the average user instead of just the IT professionals. Regards, Greg ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS: unreliable for professional usage?
Uwe Dippel wrote: We have seen some unfortunate miscommunication here, and misinterpretation. This extends into differences of culture. One of the vocal person in here is surely not 'Anti-xyz'; rather I sense his intense desire to further the progress by pointing his finger to some potential wounds. I really don't have a dog in this fight but I think what we've seen here is the behavior of a person who is too lazy to read the manual, unable to understand the technology they are working with, and unwilling to face the consequences of their own behavior. As the Solaris user base increases though, the number of people like this will increase. The general population do not read the manuals nor do they care how the magic box works, they just want it to work. This is entirely appropriate for a business user who is using the computer as a means to an end. They have their area of expertise, which isn't computers. Of course, it really isn't appropriate for a system administrator so I can't generate a lot of sympathy for DE personally, especially after the manner in which he has behaved in this thread. Turning Solaris into something that can be used with the same amount of thought as a toaster is one of the challenges facing the Sun and the community in the future. Designing guards to prevent the ignorant from harming themselves is a challenge (see quote below). There are 2 things that are infinite in this world, the universe and human stupidity. I'm not sure about the first one - Albert Einstein Regards, Greg ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Bad sectors arises - discs differ in size - trouble?
Orvar Korvar wrote: Ok. Just to confirm: A modern disk has already some spare capacity which is not normally utilized by ZFS, UFS, etc. If the spare capacity is finished, then the disc should be replaced. Yup, that is the case. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
