[Gluster-devel] Monitoring and acting on LVM thin-pool consumption
Recently I have been implementing "volume clone" support in Heketi. This uses the snapshot+clone functionality from Gluster. In order to create snapshots and clone them, it is required to use LVM thin-pools on the bricks. This is where my current problem originates When there are cloned volumes, the bricks of these volumes use the same thin-pool as the original bricks. This makes sense, and allows cloning to be really fast! There is no need to copy data from one brick to a new one, the thin-pool provides copy-on-write semantics. Unfortunately it can be rather difficult to estimate how large the thin-pool should be when the initial Gluster Volume is created. Over-allocation is likely needed, but by how much? It may not be clear how many clones there will be made, nor how much % of data will change on each of the clones. A wrong estimate can easily cause the thin-pool to become full. When that happens, the filesystem on the bricks will go readonly. Mounting the filesystem read-writable may not be possible at all. I've even seen /dev entries for the LV getting removed. This makes for a horrible Gluster experience, and it can be tricky to recover from it. In order to make thin-provisioning more stable in Gluster, I would like to see integrated monitoring of (thin) LVs and some form of acting on crucial events. One idea would be to make the Gluster Volume read-only when it detects that a brick is almost out-of-space. This is close to what local filesystems do when their block-device is having issues. The 'dmeventd' process already monitors LVM, and by default writes to 'dmesg'. Checking dmesg for warnings is not really a nice solution, so maybe we should write a plugin for dmeventd. Possibly something exists already what we can use, or take inspiration from. Please provide ideas, thoughts and any other comments. Thanks! Niels ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Monitoring and acting on LVM thin-pool consumption
I've wondered about this for a long time. Given that consumption monitoring already exists in the dmeventd thin plugin and can even trigger actions like thin pools claiming available physical extents from the VG, it certainly seems like there is an existing code structure there that we could tie into. I've just never looked into how extensible this structure is to allow for more plugins to handle the kinds of external actions we would need. A big +1 from me for the effort, at the least. -Dustin On Apr 10, 2018 5:40 AM, "Niels de Vos" wrote: Recently I have been implementing "volume clone" support in Heketi. This uses the snapshot+clone functionality from Gluster. In order to create snapshots and clone them, it is required to use LVM thin-pools on the bricks. This is where my current problem originates When there are cloned volumes, the bricks of these volumes use the same thin-pool as the original bricks. This makes sense, and allows cloning to be really fast! There is no need to copy data from one brick to a new one, the thin-pool provides copy-on-write semantics. Unfortunately it can be rather difficult to estimate how large the thin-pool should be when the initial Gluster Volume is created. Over-allocation is likely needed, but by how much? It may not be clear how many clones there will be made, nor how much % of data will change on each of the clones. A wrong estimate can easily cause the thin-pool to become full. When that happens, the filesystem on the bricks will go readonly. Mounting the filesystem read-writable may not be possible at all. I've even seen /dev entries for the LV getting removed. This makes for a horrible Gluster experience, and it can be tricky to recover from it. In order to make thin-provisioning more stable in Gluster, I would like to see integrated monitoring of (thin) LVs and some form of acting on crucial events. One idea would be to make the Gluster Volume read-only when it detects that a brick is almost out-of-space. This is close to what local filesystems do when their block-device is having issues. The 'dmeventd' process already monitors LVM, and by default writes to 'dmesg'. Checking dmesg for warnings is not really a nice solution, so maybe we should write a plugin for dmeventd. Possibly something exists already what we can use, or take inspiration from. Please provide ideas, thoughts and any other comments. Thanks! Niels ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Coverity covscan for 2018-04-10-25b8139c (master branch)
GlusterFS Coverity covscan results are available from http://download.gluster.org/pub/gluster/glusterfs/static-analysis/master/glusterfs-coverity/2018-04-10-25b8139c ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Monitoring and acting on LVM thin-pool consumption
On Tue, Apr 10, 2018 at 3:08 PM, Niels de Vos wrote: > Recently I have been implementing "volume clone" support in Heketi. This > uses the snapshot+clone functionality from Gluster. In order to create > snapshots and clone them, it is required to use LVM thin-pools on the > bricks. This is where my current problem originates > > When there are cloned volumes, the bricks of these volumes use the same > thin-pool as the original bricks. This makes sense, and allows cloning > to be really fast! There is no need to copy data from one brick to a new > one, the thin-pool provides copy-on-write semantics. > > Unfortunately it can be rather difficult to estimate how large the > thin-pool should be when the initial Gluster Volume is created. > Over-allocation is likely needed, but by how much? It may not be clear > how many clones there will be made, nor how much % of data will change > on each of the clones. > > A wrong estimate can easily cause the thin-pool to become full. When > that happens, the filesystem on the bricks will go readonly. Mounting > the filesystem read-writable may not be possible at all. I've even seen > /dev entries for the LV getting removed. This makes for a horrible > Gluster experience, and it can be tricky to recover from it. > > In order to make thin-provisioning more stable in Gluster, I would like > to see integrated monitoring of (thin) LVs and some form of acting on > crucial events. One idea would be to make the Gluster Volume read-only > when it detects that a brick is almost out-of-space. This is close to > what local filesystems do when their block-device is having issues. > > The 'dmeventd' process already monitors LVM, and by default writes to > 'dmesg'. Checking dmesg for warnings is not really a nice solution, so > maybe we should write a plugin for dmeventd. Possibly something exists > already what we can use, or take inspiration from. > > Please provide ideas, thoughts and any other comments. Thanks! > For the oVirt-Gluster integration, where gluster volumes are managed and consumed as VM image store by oVirt - a feature was added to monitor and report guaranteed capacity for bricks as opposed to the reported size when created on thin-provisioned LVs/vdo devices. The feature page provide some details - https://ovirt.org/develop/release-management/features/gluster/gluster-multiple-bricks-per-storage/. Also, adding Denis, the feature owner. Niels > ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Release 3.12.8: Scheduled for the 12th of April
Hi, It's time to prepare the 3.12.8 release, which falls on the 10th of each month, and hence would be 12-04-2018 this time around. This mail is to call out the following, 1) Are there any pending *blocker* bugs that need to be tracked for 3.12.7? If so mark them against the provided tracker [1] as blockers for the release, or at the very least post them as a response to this mail 2) Pending reviews in the 3.12 dashboard will be part of the release, *iff* they pass regressions and have the review votes, so use the dashboard [2] to check on the status of your patches to 3.12 and get these going 3) I have made checks on what went into 3.10 post 3.12 release and if these fixes are already included in 3.12 branch, then status on this is *green* as all fixes ported to 3.10, are ported to 3.12 as well. @Mlind IMO https://review.gluster.org/19659 is like a minor feature to me. Can please provide a justification for why it need to include in 3.12 stable release? And please rebase the change as well @Raghavendra The smoke failed for https://review.gluster.org/#/c/19818/. Can please check the same? Thanks, Jiffin [1] Release bug tracker: https://bugzilla.redhat.com/show_bug.cgi?id=glusterfs-3.12.8 [2] 3.12 review dashboard: https://review.gluster.org/#/projects/glusterfs,dashboards/dashboard:3-12-dashboard ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Jenkins upgrade today
Hello folks, There's a Jenkins security fix scheduled to be released today. This will most likely happen in the morning EDT. The Jenkins team has not specified a time. When we're ready for an upgrade, I'll cancel all running jobs and re-trigger them at te end of the upgrade. The downtime should be less than 15 mins. Please bear with us as we continue to ensure that build.gluster.org has the latest security fixes. -- nigelb ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
