Re: [zfs-discuss] ZFS Fragmentation issue - examining the ZIL

On Wed, Aug 03, 2011 at 12:32:56PM -0700, Brandon High wrote: > On Mon, Aug 1, 2011 at 4:27 PM, Daniel Carosone wrote: > > The other thing that can cause a storm of tiny IOs is dedup, and this > > effect can last long after space has been freed and/or dedup turned > > off, until all the blocks cor

Re: [zfs-discuss] ZFS Fragmentation issue - examining the ZIL

On Mon, Aug 1, 2011 at 4:27 PM, Daniel Carosone wrote: > The other thing that can cause a storm of tiny IOs is dedup, and this > effect can last long after space has been freed and/or dedup turned > off, until all the blocks corresponding to DDT entries are rewritten. > I wonder if this was involv

Re: [zfs-discuss] ZFS Fragmentation issue - examining the ZIL

On Mon, Aug 01, 2011 at 03:10:28PM -0700, Richard Elling wrote: > On Aug 1, 2011, at 2:16 PM, Neil Perrin wrote: > > > In general the blogs conclusion is correct . When file systems get full > > there is > > fragmentation (happens to all file systems) and for ZFS the pool uses gang > > blocks of

Re: [zfs-discuss] ZFS Fragmentation issue - examining the ZIL

On Aug 1, 2011, at 2:16 PM, Neil Perrin wrote: > In general the blogs conclusion is correct . When file systems get full there > is > fragmentation (happens to all file systems) and for ZFS the pool uses gang > blocks of smaller blocks when there are insufficient large blocks. > However, the ZIL

Re: [zfs-discuss] ZFS Fragmentation issue - examining the ZIL

On Mon, Aug 1, 2011 at 2:16 PM, Neil Perrin wrote: > In general the blogs conclusion is correct . When file systems get full > there is > fragmentation (happens to all file systems) and for ZFS the pool uses gang > blocks of smaller blocks when there are insufficient large blocks. The blog doesn'

Re: [zfs-discuss] ZFS Fragmentation issue - examining the ZIL

In general the blogs conclusion is correct . When file systems get full there is fragmentation (happens to all file systems) and for ZFS the pool uses gang blocks of smaller blocks when there are insufficient large blocks. However, the ZIL never allocates or uses gang blocks. It directly allocate

[zfs-discuss] ZFS Fragmentation issue - examining the ZIL

Hello, One of my coworkers was sent the following explanation from Oracle as to why one of backup systems was conducting a scrub so slow. I figured I would share it with the group. http://wildness.espix.org/index.php?post/2011/06/09/ZFS-Fragmentation-issue-examining-the-ZIL PS: Thought it wa

Re: [zfs-discuss] zfs fragmentation

unday, August 09, 2009 12:14 AM To: zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] zfs fragmentation On Sat, 2009-08-08 at 15:20, Bob Friesenhahn wrote: > A SSD slog backed by a SAS 15K JBOD array should perform much better > than a big iSCSI LUN. Now...yes. We implemented this

Re: [zfs-discuss] zfs fragmentation

Ed Spencer wrote: I don't know of any reason why we can't turn 1 backup job per filesystem into say, up to say , 26 based on the cyrus file and directory structure. No reason whatsoever. Sometimes the more the better as per the rest of this thread. The key here is to test and tweak till you

Re: [zfs-discuss] zfs fragmentation

On Tue, Aug 11, 2009 at 11:04 PM, Richard Elling wrote: > On Aug 11, 2009, at 7:39 AM, Ed Spencer wrote: > >> I suspect that if we 'rsync' one of these filesystems to a second >> server/pool  that we would also see a performance increase equal to what >> we see on the development server. (I don't k

Re: [zfs-discuss] zfs fragmentation

I don't know of any reason why we can't turn 1 backup job per filesystem into say, up to say , 26 based on the cyrus file and directory structure. The cyrus file and directory structure is designed with users located under the directories A,B,C,D,etc to deal with the millions of little files issue

Re: [zfs-discuss] zfs fragmentation

On Tue, Aug 11, 2009 at 9:39 AM, Ed Spencer wrote: > We backup 2 filesystems on tuesday, 2 filesystems on thursday, and 2 on > saturday. We backup to disk and then clone to tape. Our backup people > can only handle doing 2 filesystems per night. > > Creating more filesystems to increase the paralle

Re: [zfs-discuss] zfs fragmentation

On Tue, 2009-08-11 at 14:56, Scott Lawson wrote: > > Also, is atime on? > Turning atime off may make a big difference for you. It certainly does > for Sun Messaging server. > Maybe worth doing and reposting result? Yes. All these results were attained with atime=off. We made that change on all t

Re: [zfs-discuss] zfs fragmentation

On Tue, 2009-08-11 at 08:04 -0700, Richard Elling wrote: > On Aug 11, 2009, at 7:39 AM, Ed Spencer wrote: > > I suspect that if we 'rsync' one of these filesystems to a second > > server/pool that we would also see a performance increase equal to > > what > > we see on the development server. (I

Re: [zfs-discuss] zfs fragmentation

On Aug 11, 2009, at 1:21 PM, Ed Spencer wrote: Concurrency/Parallelism testing. I have 6 different filesystems populated with email data on our mail development server. I rebooted the server before beginning the tests. The server is a T2000 (sun4v) machine so its ideally suited for this type of

Re: [zfs-discuss] zfs fragmentation

Concurrency/Parallelism testing. I have 6 different filesystems populated with email data on our mail development server. I rebooted the server before beginning the tests. The server is a T2000 (sun4v) machine so its ideally suited for this type of testing. The test was to tar (to /dev/null) each o

Re: [zfs-discuss] zfs fragmentation

Richard Elling wrote: On Aug 11, 2009, at 7:39 AM, Ed Spencer wrote: On Tue, 2009-08-11 at 07:58, Alex Lam S.L. wrote: At a first glance, your production server's numbers are looking fairly similar to the "small file workload" results of your development server. I thought you were saying t

Re: [zfs-discuss] zfs fragmentation

On Tue, August 11, 2009 10:39, Ed Spencer wrote: > I suspect that if we 'rsync' one of these filesystems to a second > server/pool that we would also see a performance increase equal to what > we see on the development server. (I don't know how zfs send a receive Rsync has to traverse the entire

Re: [zfs-discuss] zfs fragmentation

On Aug 11, 2009, at 7:39 AM, Ed Spencer wrote: On Tue, 2009-08-11 at 07:58, Alex Lam S.L. wrote: At a first glance, your production server's numbers are looking fairly similar to the "small file workload" results of your development server. I thought you were saying that the development ser

Re: [zfs-discuss] zfs fragmentation

On Tue, 2009-08-11 at 07:58, Alex Lam S.L. wrote: > At a first glance, your production server's numbers are looking fairly > similar to the "small file workload" results of your development > server. > > I thought you were saying that the development server has faster performance? The developmen

Re: [zfs-discuss] zfs fragmentation

On Tue, Aug 11, 2009 at 7:33 AM, Ed Spencer wrote: > I've come up with a better name for the concept of file and directory > fragmentation which is, "Filesystem Entropy". Where, over time, an > active and volitile filesystem moves from an organized state to a > disorganized state resulting in back

Re: [zfs-discuss] zfs fragmentation

At a first glance, your production server's numbers are looking fairly similar to the "small file workload" results of your development server. I thought you were saying that the development server has faster performance? Alex. On Tue, Aug 11, 2009 at 1:33 PM, Ed Spencer wrote: > I've come up w

Re: [zfs-discuss] zfs fragmentation

I've come up with a better name for the concept of file and directory fragmentation which is, "Filesystem Entropy". Where, over time, an active and volitile filesystem moves from an organized state to a disorganized state resulting in backup difficulties. Here are some stats which illustrate the

Re: [zfs-discuss] zfs fragmentation

On Sat, Aug 8, 2009 at 20:20, Ed Spencer wrote: > > On Sat, 2009-08-08 at 08:14, Mattias Pantzare wrote: > >> Your scalability problem may be in your backup solution. > We've eliminated the backup system as being involved with the > performance issues. > > The servers are Solaris 10 with the OS on

Re: [zfs-discuss] zfs fragmentation

On Sat, 8 Aug 2009, Ed Spencer wrote: r/sw/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 11.9 43.0 528.9 1972.8 0.0 2.10.0 38.9 0 31 c4t60A98000433469764E4A2D456A644A74d0 17.0 19.6 496.9 1499.0 0.0 1.40.0 38.8 0 39 c4t60A98000433469764E4A2D456A69657

Re: [zfs-discuss] zfs fragmentation

On Aug 8, 2009, at 5:02 AM, Ed Spencer wrote: On Fri, 2009-08-07 at 19:33, Richard Elling wrote: This is very unlikely to be a "fragmentation problem." It is a scalability problem and there may be something you can do about it in the short term. You could be right. Out test mail server co

Re: [zfs-discuss] zfs fragmentation

On Sat, 2009-08-08 at 15:05, Mike Gerdts wrote: > On Sat, Aug 8, 2009 at 12:51 PM, Ed Spencer wrote: > > > > On Sat, 2009-08-08 at 09:17, Bob Friesenhahn wrote: > >> Many of us here already tested our own systems and found that under > >> some conditions ZFS was offering up only 30MB/second for bu

Re: [zfs-discuss] zfs fragmentation

On Sat, 2009-08-08 at 16:09, Mike Gerdts wrote: > Right... but ZFS doesn't understand your application. The reason that > a file system would put files that are in the same directory in the > same general area on a disk is to minimize seek time. I would argue > that seek time doesn't matter a w

Re: [zfs-discuss] zfs fragmentation

On Sat, 2009-08-08 at 15:20, Bob Friesenhahn wrote: > A SSD slog backed by a SAS 15K JBOD array should perform much better > than a big iSCSI LUN. Now...yes. We implemented this pool years ago. I believe, then, the server would crash if you had a zfs drive fail. We decided to let the netapp han

Re: [zfs-discuss] zfs fragmentation

On Sat, Aug 8, 2009 at 3:25 PM, Ed Spencer wrote: > > On Sat, 2009-08-08 at 15:12, Mike Gerdts wrote: > >> The DBA's that I know use files that are at least hundreds of >> megabytes in size.  Your problem is very different. > Yes, definitely. > > I'm relating records in a table to my small files be

Re: [zfs-discuss] zfs fragmentation

On Sat, 2009-08-08 at 15:12, Mike Gerdts wrote: > The DBA's that I know use files that are at least hundreds of > megabytes in size. Your problem is very different. Yes, definitely. I'm relating records in a table to my small files because our email system treats the filesystem as a database.

Re: [zfs-discuss] zfs fragmentation

On Sat, 8 Aug 2009, Ed Spencer wrote: On Sat, 2009-08-08 at 09:17, Bob Friesenhahn wrote: Enterprise storage should work fine without needing to run a tool to optimize data layout or repair the filesystem. Well designed software uses an approach which does not unravel through use. H, th

Re: [zfs-discuss] zfs fragmentation

On Sat, Aug 8, 2009 at 3:02 PM, Ed Spencer wrote: > > On Sat, 2009-08-08 at 09:17, Bob Friesenhahn wrote: > >> Enterprise storage should work fine without needing to run a tool to >> optimize data layout or repair the filesystem.  Well designed software >> uses an approach which does not unravel th

Re: [zfs-discuss] zfs fragmentation

On Sat, Aug 8, 2009 at 12:51 PM, Ed Spencer wrote: > > On Sat, 2009-08-08 at 09:17, Bob Friesenhahn wrote: >> Many of us here already tested our own systems and found that under >> some conditions ZFS was offering up only 30MB/second for bulk data >> reads regardless of how exotic our storage pool

Re: [zfs-discuss] zfs fragmentation

On Sat, 2009-08-08 at 09:17, Bob Friesenhahn wrote: > Enterprise storage should work fine without needing to run a tool to > optimize data layout or repair the filesystem. Well designed software > uses an approach which does not unravel through use. H, this is counter to my understanding.

Re: [zfs-discuss] zfs fragmentation

On Sat, 2009-08-08 at 08:14, Mattias Pantzare wrote: > Your scalability problem may be in your backup solution. We've eliminated the backup system as being involved with the performance issues. The servers are Solaris 10 with the OS on UFS filesystems. (In zfs terms, the pool is old/mature). So

Re: [zfs-discuss] zfs fragmentation

On Sat, 2009-08-08 at 09:17, Bob Friesenhahn wrote: > Many of us here already tested our own systems and found that under > some conditions ZFS was offering up only 30MB/second for bulk data > reads regardless of how exotic our storage pool and hardware was. Just so we are using the same units

Re: [zfs-discuss] zfs fragmentation

On Sat, 8 Aug 2009, Ed Spencer wrote: What is the point of a filesystem the can grow to such a huge size and not have functionality built in to optimize data layout? Real world implementations of filesystems that are intended to live for years/decades need this functionality, don't they? Ente

Re: [zfs-discuss] zfs fragmentation

>> > Adding another pool and copying all/some data over to it would only >> > a short term solution. >> >> I'll have to disagree. > > What is the point of a filesystem the can grow to such a huge size and > not have functionality built in to optimize data layout?  Real world > implementations of fi

Re: [zfs-discuss] zfs fragmentation

On Fri, 2009-08-07 at 19:33, Richard Elling wrote: > This is very unlikely to be a "fragmentation problem." It is a > scalability problem > and there may be something you can do about it in the short term. You could be right. Out test mail server consists of the exact same design, same hardwa

Re: [zfs-discuss] zfs fragmentation

On Aug 7, 2009, at 2:29 PM, Ed Spencer wrote: Let me give a real life example of what I believe is a fragmented zfs pool. Currently the pool is 2 terabytes in size (55% used) and is made of 4 san luns (512gb each). The pool has never gotten close to being full. We increase the size of th

Re: [zfs-discuss] zfs fragmentation

Let me give a real life example of what I believe is a fragmented zfs pool. Currently the pool is 2 terabytes in size (55% used) and is made of 4 san luns (512gb each). The pool has never gotten close to being full. We increase the size of the pool by adding 2 512gb luns about once a year or so

Re: [zfs-discuss] zfs fragmentation

On Fri, 7 Aug 2009, Scott Meilicke wrote: Bob, since the ZIL is used always, whether a separate device or not, won't writes to a system without a separate ZIL also be written as intelligently as with a separate ZIL? I don't know the answer to that. Perhaps there is no current advantage. T

Re: [zfs-discuss] zfs fragmentation

On 08/07/09 10:54, Scott Meilicke wrote: ZFS absolutely observes synchronous write requests (e.g. by NFS or a database). The synchronous write requests do not benefit from the long write aggregation delay so the result may not be written as ideally as ordinary write requests. Recently zfs has

Re: [zfs-discuss] zfs fragmentation

> ZFS absolutely observes synchronous write requests (e.g. by NFS or a > database). The synchronous write requests do not benefit from the > long write aggregation delay so the result may not be written as > ideally as ordinary write requests. Recently zfs has added support > for using a SSD as

Re: [zfs-discuss] zfs fragmentation

On Thu, 6 Aug 2009, Hua wrote: 1. Due to the COW nature of zfs, files on zfs are more tender to be fragmented comparing to traditional file system. Is this statement correct? Yes and no. Fragmentation is a complex issue. ZFS uses 128K data blocks by default whereas other filesystems typica

[zfs-discuss] zfs fragmentation

1. Due to the COW nature of zfs, files on zfs are more tender to be fragmented comparing to traditional file system. Is this statement correct? 2. If so, common understanding is that fragmentation cause perform degradation, will zfs or to what extend zfs performance is affected by the fragmentat

Re: [zfs-discuss] ZFS fragmentation with MySQL databases

t. johnson wrote: >>> One would expect so, yes. But the usefulness of this is limited to the >>> cases where the entire working set will fit into an SSD cache. >>> >> Not entirely out of the question. SSDs can be purchased today >> with more than 500 GBytes in a 2.5" form factor. One or more of >>

Re: [zfs-discuss] ZFS fragmentation with MySQL databases

>> >> One would expect so, yes. But the usefulness of this is limited to the cases >> where the entire working set will fit into an SSD cache. >> > > Not entirely out of the question. SSDs can be purchased today > with more than 500 GBytes in a 2.5" form factor. One or more of > these would make a

Re: [zfs-discuss] ZFS fragmentation with MySQL databases

Luke Lonergan wrote: >> Actually, it does seem to work quite >> well when you use a read optimized >> SSD for the L2ARC. In that case, >> "random" read workloads have very >> fast access, once the cache is warm. >> > > One would expect so, yes. But the usefulness of this is limited to the ca

Re: [zfs-discuss] ZFS fragmentation with MySQL databases

On Sun, 23 Nov 2008, Bob Netherton wrote: >> This argument can be proven by basic statistics without need to resort >> to actual testing. > > Mathematical proof <> reality of how things end up getting used. Right. That is a good thing since otherwise the technologies that Sun has recently deplo

Re: [zfs-discuss] ZFS fragmentation with MySQL databases

> This argument can be proven by basic statistics without need to resort > to actual testing. Mathematical proof <> reality of how things end up getting used. > Luckily, most data access is not completely random in nature. Which was my point exactly. I've never seen a purely mathematical mod

Re: [zfs-discuss] ZFS fragmentation with MySQL databases

On Sat, 22 Nov 2008, Bob Netherton wrote: > >> In other words, for random access across a working set larger (by >> say X%) than the SSD-backed L2 ARC, the cache is useless. This >> should asymptotically approach truth as X grows and experience >> shows that X=200% is where it's about 99% true

Re: [zfs-discuss] ZFS fragmentation with MySQL databases

> In other words, for random access across a working set larger (by say X%) > than the SSD-backed L2 ARC, the cache is useless. This should asymptotically > approach truth as X grows and experience shows that X=200% is where it's > about 99% true. > Ummm, before we throw around phrases like

Re: [zfs-discuss] ZFS fragmentation with MySQL databases

ndexed access alongside heavy analytical 'update rarely if ever' kind of workloads. - Luke - Original Message - From: [EMAIL PROTECTED] <[EMAIL PROTECTED]> To: Luke Lonergan Cc: [EMAIL PROTECTED] <[EMAIL PROTECTED]>; zfs-discuss@opensolaris.org Sent: Sat Nov 22 2

Re: [zfs-discuss] ZFS fragmentation with MySQL databases

ROTECTED]> > To: zfs-discuss@opensolaris.org > Sent: Sat Nov 22 16:43:53 2008 > Subject: Re: [zfs-discuss] ZFS fragmentation with MySQL databases > > Kees Nuyt wrote: > >> My explanation would be: Whenever a block within a file >> changes, zfs has to write it at

Re: [zfs-discuss] ZFS fragmentation with MySQL databases

On Sun, 23 Nov 2008, Tamer Embaby wrote: >> That is the trade-off between "always consistent" and >> "fast". >> > Well, does that mean ZFS is not best suited for database engines as > underlying filesystem? With databases it will always be fragmented, > hence slow performance? Assuming that the

Re: [zfs-discuss] ZFS fragmentation with MySQL databases

EMAIL PROTECTED]> To: zfs-discuss@opensolaris.org Sent: Sat Nov 22 16:43:53 2008 Subject: Re: [zfs-discuss] ZFS fragmentation with MySQL databases Kees Nuyt wrote: > My explanation would be: Whenever a block within a file > changes, zfs has to write it at another location ("copy on >

Re: [zfs-discuss] ZFS fragmentation with MySQL databases

Kees Nuyt wrote: > My explanation would be: Whenever a block within a file > changes, zfs has to write it at another location ("copy on > write"), so the previous version isn't immediately lost. > > Zfs will try to keep the new version of the block close to > the original one, but after several cha

Re: [zfs-discuss] ZFS fragmentation with MySQL databases

[Default] On Fri, 21 Nov 2008 17:20:48 PST, Vincent Kéravec <[EMAIL PROTECTED]> wrote: > I just try ZFS on one of our slave and got some really > bad performance. > > When I start the server yesterday, it was able to keep > up with the main server without problem but after two > days of consecuti

[zfs-discuss] ZFS fragmentation with MySQL databases

I just try ZFS on one of our slave and got some really bad performance. When I start the server yesterday, it was able to keep up with the main server without problem but after two days of consecutive run the server is crushed by IO. After running the dtrace script iopattern, I notice that the

Re: [zfs-discuss] ZFS fragmentation

Lance, This could be bug#*6596237 Stop looking and start ganging .* The fix is in progress and Victor Latushkin is working on it. We have an IDR based on the patches 127127-11/127128-11 which has the first cut of the fix. You could raise an escalation

Re: [zfs-discuss] ZFS fragmentation

Any progress on a defragmentation utility? We appear to be having a severe fragmentation problem on an X4500, vanilla S10U4, no additional patches. 500GB disks in 4 x 11 disk RAIDZ2 vdevs. It hit 97% full and fell off a cliff...about 50KB/sec on writes. Deleting files so the zpool is at 92%

Re: [zfs-discuss] ZFS fragmentation

Matty wrote: Howdy, I have seen a number of folks run into issues due to ZFS file system fragmentation, and was curious if anyone on team ZFS is working on this issue? Would it be possible to share with the list any changes that will be made to to help address fragmentation problems? We have s

[zfs-discuss] ZFS fragmentation

Howdy, I have seen a number of folks run into issues due to ZFS file system fragmentation, and was curious if anyone on team ZFS is working on this issue? Would it be possible to share with the list any changes that will be made to to help address fragmentation problems? Thanks, - Ryan -- UNIX A