Re: btrfs raid5
On Oct 22, 2013, Duncan 1i5t5.dun...@cox.net wrote: the quick failure should they try raid56 in its current state simply alerts them to the problem they already had. What quick failure? There's no such thing in place AFAIK. It seems to do all the work properly, the limitations in the current implementation will only show up when an I/O error kicks in. I can't see any indication, in existing announcements, that recovery from I/O errors in raid56 is missing, let alone that it's so utterly and completely broken that it will freeze the entire filesystem and require a forced reboot to unmount the filesystem and make any other data in it accessible again. That's far, far worse than the general state of btrfs, and that's not a documented limitation of raid56, so how would someone be expected to know about it? It certainly isn't obvious by having a cursory look at the code either. -- Alexandre Oliva, freedom fighterhttp://FSFLA.org/~lxoliva/ You must be the change you wish to see in the world. -- Gandhi Be Free! -- http://FSFLA.org/ FSF Latin America board member Free Software Evangelist Red Hat Brazil Compiler Engineer -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: swapfile on btrfs, temporary solution for wiki
Is the NFS over network code upstream ? On 10/25/13, Hugo Mills h...@carfax.org.uk wrote: On Fri, Oct 25, 2013 at 04:35:46PM +0600, Roman Mamedov wrote: On Thu, 24 Oct 2013 23:52:01 +0300 Timofey Titovets nefelim...@gmail.com wrote: Hello, i suggest temporary solution to use swap file under btrfs. I test it, and it work good. I invent simple the way, how create and using swap file, just see following sh code: swapfile=$(losetup -f) #free loop device truncate -s 8G /swap #create 8G sparse swap file losetup $swapfile /swap #mount file to loop mkswap $swapfile swapon $swapfile i just adding this to rc.local and this work good. May be, add it to btrfs Wiki as temporary solution to using swap file? I always thought Btrfs does not allow swap files on purpose, because it is not deadlock-proof when used in the swapping context. It's more that the current swap interface is based on device+block list, and if you balance a filesystem, the blocks for a file move -- but there's no way of telling the swap code to cope with that. Imagine you try swapping out pages to free up some memory, and in the process Btrfs needs to allocate some memory to actually perform the write, the kernel says Sure, but for that we need to swap out some more pages... You see where that goes. Same issue is possible with swap on other complex filesystems an example being networked ones like NFS and SMB/CIFS. The network filesystems have a similar problem as btrfs -- they don't export devices, and you don't get direct access to the low-level blocks under the FS, so the swap code can't deal with it. That said, there's been a lot of work recently on getting swap-over-NFS to work properly -- effectively giving a new interface for the swap code that doesn't rely on direct mapping to device blocks. That new interface gives us the minimal external infrastructure necessary to consider doing swapfiles on btrfs. It might work for some time (or even work 99% of time), but you still may be endangering your system to a possibility of a lock-up, and certainly adding that to any Wiki/FAQ/website as the solution might not be the best choice. The deadlock situation is dealt with by adding a flag to the memory allocator in the swap-critical paths, which says you're not allowed to swap anything when you make the allocation. That at least allows the memory allocation to fail (hopefully gracefully) without deadlocking. I suspect that this is also part of the swap-on-NFS work -- adding that flag everywhere necessary. Hugo. -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- He's playing Schubert. I think Schubert is losing. --- -- Karim Allah Ahmed. LinkedIn http://eg.linkedin.com/pub/karim-allah-ahmed/13/829/550/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: swapfile on btrfs, temporary solution for wiki
karim.allah.ah...@gmail.com posted on Sat, 26 Oct 2013 10:30:18 +0100 as excerpted: Is the swap over NFS code upstream ? ?esrever ni txet referp uoy od ;tsop-pot t'nod esaelP (Please don't top-post; do you prefer text in reverse?) Quote the context you need so your question makes sense and reply below it, and you're more likely to get useful answers as it'll be less work to reply properly. Here's how it should have looked, now quoted another level for my reply: That said, there's been a lot of work recently on getting swap-over-NFS to work properly -- effectively giving a new interface for the swap code that doesn't rely on direct mapping to device blocks. That new interface gives us the minimal external infrastructure necessary to consider doing swapfiles on btrfs. Is the swap over NFS code upstream ? The answer is yes, I've definitely seen commit comments about swap over NFS on recent commits, so the work is indeed going in, tho it may well still be a WIP. (I don't use NFS here and in fact with 8-16 gigs RAM common these days, I don't use swap so much any more either and often don't even have the kernel swap option on for my current configs unless I'm using it for suspend-to-disk, so I've not followed the issue closely enough to know current in-kernel status, but based on the commits I've happened across, yes, the work is certainly going in, if the feature isn't already there and working in general and they're simply tweaking it now.) -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch 3/7] btrfs: Add per-super attributes to sysfs
Hi Jeff,
On Tue, Sep 10, 2013 at 7:24 AM, Jeff Mahoney je...@suse.com wrote:
This patch adds per-super attributes to sysfs.
It doesn't publish any attributes yet, but does the proper lifetime
handling as well as the basic infrastructure to add new attributes.
Signed-off-by: Jeff Mahoney je...@suse.com
---
fs/btrfs/ctree.h |2 +
fs/btrfs/super.c | 13 +++-
fs/btrfs/sysfs.c | 58
+++
fs/btrfs/sysfs.h | 19 ++
4 files changed, 91 insertions(+), 1 deletion(-)
--- a/fs/btrfs/ctree.h 2013-09-10 00:09:12.990087784 -0400
+++ b/fs/btrfs/ctree.h 2013-09-10 00:09:35.521794520 -0400
@@ -3694,6 +3694,8 @@ int btrfs_defrag_leaves(struct btrfs_tra
/* sysfs.c */
int btrfs_init_sysfs(void);
void btrfs_exit_sysfs(void);
+int btrfs_sysfs_add_one(struct btrfs_fs_info *fs_info);
+void btrfs_sysfs_remove_one(struct btrfs_fs_info *fs_info);
/* xattr.c */
ssize_t btrfs_listxattr(struct dentry *dentry, char *buffer, size_t size);
--- a/fs/btrfs/super.c 2013-09-10 00:09:12.994087730 -0400
+++ b/fs/btrfs/super.c 2013-09-10 00:09:35.525794464 -0400
@@ -301,6 +301,8 @@ void __btrfs_panic(struct btrfs_fs_info
static void btrfs_put_super(struct super_block *sb)
{
+ btrfs_sysfs_remove_one(btrfs_sb(sb));
+
(void)close_ctree(btrfs_sb(sb)-tree_root);
/* FIXME: need to fix VFS to return error? */
/* AV: return it _where_? -put_super() can be triggered by any
number
@@ -1143,8 +1145,17 @@ static struct dentry *btrfs_mount(struct
}
root = !error ? get_default_root(s, subvol_objectid) : ERR_PTR(error);
- if (IS_ERR(root))
+ if (IS_ERR(root)) {
deactivate_locked_super(s);
+ return root;
+ }
+
+ error = btrfs_sysfs_add_one(fs_info);
+ if (error) {
+ dput(root);
+ deactivate_locked_super(s);
+ return ERR_PTR(error);
+ }
return root;
--- a/fs/btrfs/sysfs.c 2013-09-10 00:09:13.002087628 -0400
+++ b/fs/btrfs/sysfs.c 2013-09-10 00:09:49.501616538 -0400
@@ -61,6 +61,64 @@ static struct attribute *btrfs_supp_feat
NULL
};
+static struct attribute *btrfs_attrs[] = {
+ NULL,
+};
+
+static void btrfs_fs_info_release(struct kobject *kobj)
+{
+ struct btrfs_fs_info *fs_info;
+ fs_info = container_of(kobj, struct btrfs_fs_info, super_kobj);
+ complete(fs_info-kobj_unregister);
+}
+
+static ssize_t btrfs_attr_show(struct kobject *kobj,
+ struct attribute *attr, char *buf)
+{
+ struct btrfs_attr *a = container_of(attr, struct btrfs_attr, attr);
+ struct btrfs_fs_info *fs_info;
+ fs_info = container_of(kobj, struct btrfs_fs_info, super_kobj);
+
+ return a-show ? a-show(a, fs_info, buf) : 0;
+}
+
+static ssize_t btrfs_attr_store(struct kobject *kobj,
+ struct attribute *attr,
+ const char *buf, size_t len)
+{
+ struct btrfs_attr *a = container_of(attr, struct btrfs_attr, attr);
+ struct btrfs_fs_info *fs_info;
+ fs_info = container_of(kobj, struct btrfs_fs_info, super_kobj);
+
+ return a-store ? a-store(a, fs_info, buf, len) : 0;
+}
+
+static const struct sysfs_ops btrfs_attr_ops = {
+ .show = btrfs_attr_show,
+ .store = btrfs_attr_store,
+};
+
+static struct kobj_type btrfs_ktype = {
+ .default_attrs = btrfs_attrs,
+ .sysfs_ops = btrfs_attr_ops,
+ .release= btrfs_fs_info_release,
+};
+
+int btrfs_sysfs_add_one(struct btrfs_fs_info *fs_info)
+{
+ init_completion(fs_info-kobj_unregister);
+ fs_info-super_kobj.kset = btrfs_kset;
+ return kobject_init_and_add(fs_info-super_kobj, btrfs_ktype, NULL,
+ %pU, fs_info-fsid);
+}
+
+void btrfs_sysfs_remove_one(struct btrfs_fs_info *fs_info)
+{
+ kobject_del(fs_info-super_kobj);
Is there a reason for this explicit call? The last kobject_put will do
this automatically, no?
+ kobject_put(fs_info-super_kobj);
+ wait_for_completion(fs_info-kobj_unregister);
+}
+
static void btrfs_supp_feat_release(struct kobject *kobj)
{
complete(btrfs_feat-f_kobj_unregister);
--- a/fs/btrfs/sysfs.h 2013-09-10 00:09:13.002087628 -0400
+++ b/fs/btrfs/sysfs.h 2013-09-10 00:09:35.525794464 -0400
@@ -8,6 +8,24 @@ enum btrfs_feature_set {
FEAT_MAX
};
+struct btrfs_attr {
+ struct attribute attr;
+ ssize_t (*show)(struct btrfs_attr *, struct btrfs_fs_info *, char *);
+ ssize_t (*store)(struct btrfs_attr *, struct btrfs_fs_info *,
+const char *, size_t);
+};
+
+#define __INIT_BTRFS_ATTR(_name, _mode, _show, _store) \
+{ \
+
Re: [patch 3/7] btrfs: Add per-super attributes to sysfs
On 10/26/13, 3:00 PM, Alex Lyakas wrote:
Hi Jeff,
On Tue, Sep 10, 2013 at 7:24 AM, Jeff Mahoney je...@suse.com wrote:
This patch adds per-super attributes to sysfs.
It doesn't publish any attributes yet, but does the proper lifetime
handling as well as the basic infrastructure to add new attributes.
Signed-off-by: Jeff Mahoney je...@suse.com
---
fs/btrfs/ctree.h |2 +
fs/btrfs/super.c | 13 +++-
fs/btrfs/sysfs.c | 58
+++
fs/btrfs/sysfs.h | 19 ++
4 files changed, 91 insertions(+), 1 deletion(-)
--- a/fs/btrfs/ctree.h 2013-09-10 00:09:12.990087784 -0400
+++ b/fs/btrfs/ctree.h 2013-09-10 00:09:35.521794520 -0400
@@ -3694,6 +3694,8 @@ int btrfs_defrag_leaves(struct btrfs_tra
/* sysfs.c */
int btrfs_init_sysfs(void);
void btrfs_exit_sysfs(void);
+int btrfs_sysfs_add_one(struct btrfs_fs_info *fs_info);
+void btrfs_sysfs_remove_one(struct btrfs_fs_info *fs_info);
/* xattr.c */
ssize_t btrfs_listxattr(struct dentry *dentry, char *buffer, size_t size);
--- a/fs/btrfs/super.c 2013-09-10 00:09:12.994087730 -0400
+++ b/fs/btrfs/super.c 2013-09-10 00:09:35.525794464 -0400
@@ -301,6 +301,8 @@ void __btrfs_panic(struct btrfs_fs_info
static void btrfs_put_super(struct super_block *sb)
{
+ btrfs_sysfs_remove_one(btrfs_sb(sb));
+
(void)close_ctree(btrfs_sb(sb)-tree_root);
/* FIXME: need to fix VFS to return error? */
/* AV: return it _where_? -put_super() can be triggered by any number
@@ -1143,8 +1145,17 @@ static struct dentry *btrfs_mount(struct
}
root = !error ? get_default_root(s, subvol_objectid) : ERR_PTR(error);
- if (IS_ERR(root))
+ if (IS_ERR(root)) {
deactivate_locked_super(s);
+ return root;
+ }
+
+ error = btrfs_sysfs_add_one(fs_info);
+ if (error) {
+ dput(root);
+ deactivate_locked_super(s);
+ return ERR_PTR(error);
+ }
return root;
--- a/fs/btrfs/sysfs.c 2013-09-10 00:09:13.002087628 -0400
+++ b/fs/btrfs/sysfs.c 2013-09-10 00:09:49.501616538 -0400
@@ -61,6 +61,64 @@ static struct attribute *btrfs_supp_feat
NULL
};
+static struct attribute *btrfs_attrs[] = {
+ NULL,
+};
+
+static void btrfs_fs_info_release(struct kobject *kobj)
+{
+ struct btrfs_fs_info *fs_info;
+ fs_info = container_of(kobj, struct btrfs_fs_info, super_kobj);
+ complete(fs_info-kobj_unregister);
+}
+
+static ssize_t btrfs_attr_show(struct kobject *kobj,
+ struct attribute *attr, char *buf)
+{
+ struct btrfs_attr *a = container_of(attr, struct btrfs_attr, attr);
+ struct btrfs_fs_info *fs_info;
+ fs_info = container_of(kobj, struct btrfs_fs_info, super_kobj);
+
+ return a-show ? a-show(a, fs_info, buf) : 0;
+}
+
+static ssize_t btrfs_attr_store(struct kobject *kobj,
+ struct attribute *attr,
+ const char *buf, size_t len)
+{
+ struct btrfs_attr *a = container_of(attr, struct btrfs_attr, attr);
+ struct btrfs_fs_info *fs_info;
+ fs_info = container_of(kobj, struct btrfs_fs_info, super_kobj);
+
+ return a-store ? a-store(a, fs_info, buf, len) : 0;
+}
+
+static const struct sysfs_ops btrfs_attr_ops = {
+ .show = btrfs_attr_show,
+ .store = btrfs_attr_store,
+};
+
+static struct kobj_type btrfs_ktype = {
+ .default_attrs = btrfs_attrs,
+ .sysfs_ops = btrfs_attr_ops,
+ .release= btrfs_fs_info_release,
+};
+
+int btrfs_sysfs_add_one(struct btrfs_fs_info *fs_info)
+{
+ init_completion(fs_info-kobj_unregister);
+ fs_info-super_kobj.kset = btrfs_kset;
+ return kobject_init_and_add(fs_info-super_kobj, btrfs_ktype, NULL,
+ %pU, fs_info-fsid);
+}
+
+void btrfs_sysfs_remove_one(struct btrfs_fs_info *fs_info)
+{
+ kobject_del(fs_info-super_kobj);
Is there a reason for this explicit call? The last kobject_put will do
this automatically, no?
This should be the last reference, but even if it's not, it should be
removed from sysfs here. Otherwise, I suppose it's personal preference.
The call to kobject_del in kobject_release will also drop a debugging
message if kobject debugging is enabled.
-Jeff
+ kobject_put(fs_info-super_kobj);
+ wait_for_completion(fs_info-kobj_unregister);
+}
+
static void btrfs_supp_feat_release(struct kobject *kobj)
{
complete(btrfs_feat-f_kobj_unregister);
--- a/fs/btrfs/sysfs.h 2013-09-10 00:09:13.002087628 -0400
+++ b/fs/btrfs/sysfs.h 2013-09-10 00:09:35.525794464 -0400
@@ -8,6 +8,24 @@ enum btrfs_feature_set {
FEAT_MAX
};
+struct btrfs_attr {
+ struct attribute attr;
+ ssize_t (*show)(struct btrfs_attr *, struct btrfs_fs_info *, char *);
+ ssize_t (*store)(struct btrfs_attr *, struct btrfs_fs_info *,
+
Re: [PATCH] btrfs-progs: calculate disk space that a subvol could free upon delete
Hi Anand,
1) so let's say I have a subvolume and a snapshot of this subvolume.
So in this case, I will see Sole space = 0 for both of them,
correct? Because all extents (except inline ones) are shared.
2) How is this in terms of responsiveness? On a huge subvolume, we
need to iterate all the EXTENT_DATAs and then lookup their
EXTENT_ITEMs.
3) So it's kind of poor man's replacement for quota groups, correct?
I like that it's so easy to check for exclusive data, though:)
Alex.
On Fri, Sep 13, 2013 at 6:44 PM, Wang Shilong wangshilong1...@gmail.com wrote:
Hello Anand,
(This patch is for review and comments only)
This patch provides a way to know how much space can be
relinquished if when subvol /snapshot is deleted. With
this sys admin can make better judgments in managing the
filesystem when fs is near full.
I think this is really *helpful* since users can not really know how much
space(Exclusive) in a subvolume .
Thanks,
Wang
as shown below the parameter 'sole space' indicates the size
which is freed when subvol is deleted. (any other better
term for this?, pls suggest).
-
btrfs su show /btrfs/sv1
/btrfs/sv1
Name: sv1
uuid: b078ba48-d4a5-2f49-ac03-9bd1d56cc768
Parent uuid:-
Creation time: 2013-09-13 18:17:32
Object ID: 257
Generation (Gen): 18
Gen at creation:17
Parent: 5
Top Level: 5
Flags: -
Sole space: 1.56MiB
Snapshot(s):
btrfs su snap /btrfs/sv1 /btrfs/ss2
Create a snapshot of '/btrfs/sv1' in '/btrfs/ss2'
btrfs su show /btrfs/sv1
/btrfs/sv1
Name: sv1
uuid: b078ba48-d4a5-2f49-ac03-9bd1d56cc768
Parent uuid:-
Creation time: 2013-09-13 18:17:32
Object ID: 257
Generation (Gen): 19
Gen at creation:17
Parent: 5
Top Level: 5
Flags: -
Sole space: 0.00 -
Snapshot(s):
ss2
-
Signed-off-by: Anand Jain anand.j...@oracle.com
---
cmds-subvolume.c | 5 ++
utils.c | 154
+++
utils.h | 1 +
3 files changed, 160 insertions(+)
diff --git a/cmds-subvolume.c b/cmds-subvolume.c
index de246ab..2b02d66 100644
--- a/cmds-subvolume.c
+++ b/cmds-subvolume.c
@@ -809,6 +809,7 @@ static int cmd_subvol_show(int argc, char **argv)
int fd = -1, mntfd = -1;
int ret = 1;
DIR *dirstream1 = NULL, *dirstream2 = NULL;
+ u64 freeable_bytes;
if (check_argc_exact(argc, 2))
usage(cmd_subvol_show_usage);
@@ -878,6 +879,8 @@ static int cmd_subvol_show(int argc, char **argv)
goto out;
}
+ freeable_bytes = get_subvol_freeable_bytes(fd);
+
ret = 0;
/* print the info */
printf(%s\n, fullpath);
@@ -915,6 +918,8 @@ static int cmd_subvol_show(int argc, char **argv)
else
printf(\tFlags: \t\t\t-\n);
+ printf(\tSole space: \t\t%s\n,
+ pretty_size(freeable_bytes));
/* print the snapshots of the given subvol if any*/
printf(\tSnapshot(s):\n);
filter_set = btrfs_list_alloc_filter_set();
diff --git a/utils.c b/utils.c
index 22c3310..f01d580 100644
--- a/utils.c
+++ b/utils.c
@@ -2019,3 +2019,157 @@ int is_dev_excl_op_free(int fd)
ret = ioctl(fd, BTRFS_IOC_CHECK_DEV_EXCL_OPS, NULL);
return ret 0 ? ret : -errno;
}
+
+/* gets the ref count for given extent
+ * 0 = didn't find the item
+ * n = number of references
+*/
+u64 get_extent_refcnt(int fd, u64 disk_blk)
+{
+ int ret = 0, i, e;
+ struct btrfs_ioctl_search_args args;
+ struct btrfs_ioctl_search_key *sk = args.key;
+ struct btrfs_ioctl_search_header sh;
+ unsigned long off = 0;
+
+ memset(args, 0, sizeof(args));
+
+ sk-tree_id = BTRFS_EXTENT_TREE_OBJECTID;
+
+ sk-min_type = BTRFS_EXTENT_ITEM_KEY;
+ sk-max_type = BTRFS_EXTENT_ITEM_KEY;
+
+ sk-min_objectid = disk_blk;
+ sk-max_objectid = disk_blk;
+
+ sk-max_offset = (u64)-1;
+ sk-max_transid = (u64)-1;
+
+ while (1) {
+ sk-nr_items = 4096;
+
+ ret = ioctl(fd, BTRFS_IOC_TREE_SEARCH, args);
+ e = errno;
+ if (ret 0) {
+ fprintf(stderr, ERROR: search failed - %s\n,
+ strerror(e));
+ return 0;
+ }
+ if (sk-nr_items == 0)
+ break;
+
+ off = 0;
+ for (i = 0; i sk-nr_items; i++) {
+ struct btrfs_extent_item *ei;
+ u64 ref;
No space left on device, problem
Hello, I just upgraded kernel to 3.11.6 added new disk and created btrfs: # mkfs.btrfs /dev/sdb # mount -t btrfs -o compress=lzo,compress-force=lzo /dev/sdb /usr/local/mysql/data I started copying files from old disk and then I got 'No space left on device', but there is a lot of space. # df -h FilesystemSize Used Avail Use% Mounted on /dev/sdb 2.8T 110G 2.7T 4% /usr/local/mysql/data # btrfs fi show Label: none uuid: c0bfcb22-8b7c-4936-afcd-7acdf58f1d6c Total devices 1 FS bytes used 108.68GB devid1 size 2.73TB used 113.04GB path /dev/sdb # btrfs fi df /usr/local/mysql/data Data: total=111.01GB, used=108.25GB System, DUP: total=8.00MB, used=20.00KB System: total=4.00MB, used=0.00 Metadata, DUP: total=1.00GB, used=441.91MB Metadata: total=8.00MB, used=0.00 I tried balance from FAQ: # btrfs fi balance start -dusage=5 /usr/local/mysql/data Done, had to relocate 4 out of 117 chunks But it doesn't help. When I try to copy file there is still 'No space left on device'. What to do ? Regards, Igor -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs-progs: calculate disk space that a subvol could free upon delete
Thinking about this more, I believe this way of checking for exclusive
data doesn't work. When a snapshot is created, btrfs doesn't go and
explicitly increment refcount on *all* relevant EXTENT_ITEMs in the
extent tree. This way creating a snapshot would take forever for large
subvolumes. Instead, it only does that on EXTENT_ITEMs, which are
pointed by EXTENT_DATAs in the root node of the snapshottted file
tree. For rest of nodes/leafs of a file tree, an implicit tree-block
references are added (not sure if implicit is the right term) for
top tree blocks only. This is accomplished by _btrfs_mod_ref() code,
called from btrfs_copy_root() during snapshot creation flow. Snapshot
deletion code is the one that is smart enough to properly unshare
shared tree blocks with such implicit references.
What do you think?
Alex.
On Sat, Oct 26, 2013 at 10:49 PM, Alex Lyakas
alex.bt...@zadarastorage.com wrote:
Hi Anand,
1) so let's say I have a subvolume and a snapshot of this subvolume.
So in this case, I will see Sole space = 0 for both of them,
correct? Because all extents (except inline ones) are shared.
2) How is this in terms of responsiveness? On a huge subvolume, we
need to iterate all the EXTENT_DATAs and then lookup their
EXTENT_ITEMs.
3) So it's kind of poor man's replacement for quota groups, correct?
I like that it's so easy to check for exclusive data, though:)
Alex.
On Fri, Sep 13, 2013 at 6:44 PM, Wang Shilong wangshilong1...@gmail.com
wrote:
Hello Anand,
(This patch is for review and comments only)
This patch provides a way to know how much space can be
relinquished if when subvol /snapshot is deleted. With
this sys admin can make better judgments in managing the
filesystem when fs is near full.
I think this is really *helpful* since users can not really know how much
space(Exclusive) in a subvolume .
Thanks,
Wang
as shown below the parameter 'sole space' indicates the size
which is freed when subvol is deleted. (any other better
term for this?, pls suggest).
-
btrfs su show /btrfs/sv1
/btrfs/sv1
Name: sv1
uuid: b078ba48-d4a5-2f49-ac03-9bd1d56cc768
Parent uuid:-
Creation time: 2013-09-13 18:17:32
Object ID: 257
Generation (Gen): 18
Gen at creation:17
Parent: 5
Top Level: 5
Flags: -
Sole space: 1.56MiB
Snapshot(s):
btrfs su snap /btrfs/sv1 /btrfs/ss2
Create a snapshot of '/btrfs/sv1' in '/btrfs/ss2'
btrfs su show /btrfs/sv1
/btrfs/sv1
Name: sv1
uuid: b078ba48-d4a5-2f49-ac03-9bd1d56cc768
Parent uuid:-
Creation time: 2013-09-13 18:17:32
Object ID: 257
Generation (Gen): 19
Gen at creation:17
Parent: 5
Top Level: 5
Flags: -
Sole space: 0.00 -
Snapshot(s):
ss2
-
Signed-off-by: Anand Jain anand.j...@oracle.com
---
cmds-subvolume.c | 5 ++
utils.c | 154
+++
utils.h | 1 +
3 files changed, 160 insertions(+)
diff --git a/cmds-subvolume.c b/cmds-subvolume.c
index de246ab..2b02d66 100644
--- a/cmds-subvolume.c
+++ b/cmds-subvolume.c
@@ -809,6 +809,7 @@ static int cmd_subvol_show(int argc, char **argv)
int fd = -1, mntfd = -1;
int ret = 1;
DIR *dirstream1 = NULL, *dirstream2 = NULL;
+ u64 freeable_bytes;
if (check_argc_exact(argc, 2))
usage(cmd_subvol_show_usage);
@@ -878,6 +879,8 @@ static int cmd_subvol_show(int argc, char **argv)
goto out;
}
+ freeable_bytes = get_subvol_freeable_bytes(fd);
+
ret = 0;
/* print the info */
printf(%s\n, fullpath);
@@ -915,6 +918,8 @@ static int cmd_subvol_show(int argc, char **argv)
else
printf(\tFlags: \t\t\t-\n);
+ printf(\tSole space: \t\t%s\n,
+ pretty_size(freeable_bytes));
/* print the snapshots of the given subvol if any*/
printf(\tSnapshot(s):\n);
filter_set = btrfs_list_alloc_filter_set();
diff --git a/utils.c b/utils.c
index 22c3310..f01d580 100644
--- a/utils.c
+++ b/utils.c
@@ -2019,3 +2019,157 @@ int is_dev_excl_op_free(int fd)
ret = ioctl(fd, BTRFS_IOC_CHECK_DEV_EXCL_OPS, NULL);
return ret 0 ? ret : -errno;
}
+
+/* gets the ref count for given extent
+ * 0 = didn't find the item
+ * n = number of references
+*/
+u64 get_extent_refcnt(int fd, u64 disk_blk)
+{
+ int ret = 0, i, e;
+ struct btrfs_ioctl_search_args args;
+ struct btrfs_ioctl_search_key *sk = args.key;
+ struct btrfs_ioctl_search_header
Re: No space left on device, problem
Some more info, exact error message is: cp: writing ‘/usr/local/mysql/data/gbdata/parts_0015.MYI’: No space left on device cp: failed to extend ‘/usr/local/mysql/data/gbdata/parts_0015.MYI’: No space left on device Files are 2.7G - 7.7G big. On Sat, Oct 26, 2013 at 9:46 PM, Igor M igor...@gmail.com wrote: Hello, I just upgraded kernel to 3.11.6 added new disk and created btrfs: # mkfs.btrfs /dev/sdb # mount -t btrfs -o compress=lzo,compress-force=lzo /dev/sdb /usr/local/mysql/data I started copying files from old disk and then I got 'No space left on device', but there is a lot of space. # df -h FilesystemSize Used Avail Use% Mounted on /dev/sdb 2.8T 110G 2.7T 4% /usr/local/mysql/data # btrfs fi show Label: none uuid: c0bfcb22-8b7c-4936-afcd-7acdf58f1d6c Total devices 1 FS bytes used 108.68GB devid1 size 2.73TB used 113.04GB path /dev/sdb # btrfs fi df /usr/local/mysql/data Data: total=111.01GB, used=108.25GB System, DUP: total=8.00MB, used=20.00KB System: total=4.00MB, used=0.00 Metadata, DUP: total=1.00GB, used=441.91MB Metadata: total=8.00MB, used=0.00 I tried balance from FAQ: # btrfs fi balance start -dusage=5 /usr/local/mysql/data Done, had to relocate 4 out of 117 chunks But it doesn't help. When I try to copy file there is still 'No space left on device'. What to do ? Regards, Igor -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
On Oct 26, 2013, at 1:46 PM, Igor M igor...@gmail.com wrote: # mount -t btrfs -o compress=lzo,compress-force=lzo / Why do you have two compression mount options? You need to pick one of these. What to do ? Are there any kernel messages reported by dmesg at the time the copy starts and fails? What's the exact copy command you're using? Chris Murphy-- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
On Sat, Oct 26, 2013 at 11:35 PM, Chris Murphy li...@colorremedies.com wrote: On Oct 26, 2013, at 1:46 PM, Igor M igor...@gmail.com wrote: # mount -t btrfs -o compress=lzo,compress-force=lzo / Why do you have two compression mount options? You need to pick one of these. I removed one. I was thinking both were needed. What to do ? Are there any kernel messages reported by dmesg at the time the copy starts and fails? What's the exact copy command you're using? No messages. Just whem mounting: device fsid c0bfcb22-8b7c-4936-afcd- 7acdf58f1d6c devid 1 transid 622 /dev/sdb btrfs: force lzo compression btrfs: disk space caching is enabled I even added enospc_debug mount option, still no messages. I'm using simple cp command: # cp -a /mnt/old_hd/data/gbdata/* /usr/local/mysql/data/gbdata/ or for one file # cp -a /mnt/old_hd/data/gbdata/parts_0016.MYD /usr/local/mysql/data/gbdata/ cp: writing ‘/usr/local/mysql/data/gbdata/parts_0016.MYD’: No space left on device cp: failed to extend ‘/usr/local/mysql/data/gbdata/parts_0016.MYD’: No space left on device It's the same error if I try to copy for ex. with midnight commander. On Sat, Oct 26, 2013 at 11:35 PM, Chris Murphy li...@colorremedies.com wrote: On Oct 26, 2013, at 1:46 PM, Igor M igor...@gmail.com wrote: # mount -t btrfs -o compress=lzo,compress-force=lzo / Why do you have two compression mount options? You need to pick one of these. What to do ? Are there any kernel messages reported by dmesg at the time the copy starts and fails? What's the exact copy command you're using? Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
On Oct 26, 2013, at 3:53 PM, Igor M igor...@gmail.com wrote: I even added enospc_debug mount option, still no messages. If it were kernel enospc, you should have messages in dmesg. What version of btrfs progs when making the btrfs volume? cp: failed to extend ‘/usr/local/mysql/data/gbdata/parts_0016.MYD’: No space left on device Reboot with kernel parameter ignore_loglevel and retry the copy, and see if you now have anything in dmesg at the time of the copy. It's the same error if I try to copy for ex. with midnight commander. I have the same kernel version, the same mount options, and use the same cp -a on a 5.3GB file and cannot reproduce your results. Chris Murphy-- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
On Sun, Oct 27, 2013 at 12:17 AM, Chris Murphy li...@colorremedies.com wrote: On Oct 26, 2013, at 3:53 PM, Igor M igor...@gmail.com wrote: I even added enospc_debug mount option, still no messages. If it were kernel enospc, you should have messages in dmesg. What version of btrfs progs when making the btrfs volume? cp: failed to extend ‘/usr/local/mysql/data/gbdata/parts_0016.MYD’: No space left on device Reboot with kernel parameter ignore_loglevel and retry the copy, and see if you now have anything in dmesg at the time of the copy. Some more info. This files are MySQL database files. Tables contains only text so they compress well. But, if I copy some uncompressable file, for example some video than everything is ok, no error message, copy is successfull. I also tried mounting without compression (without compress-force=lzo) and in this case no error is reported. So it seems it's something with compression ? I'll try rebooting with ignore_loglevel parameter. It's the same error if I try to copy for ex. with midnight commander. I have the same kernel version, the same mount options, and use the same cp -a on a 5.3GB file and cannot reproduce your results. Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
On Sun, Oct 27, 2013 at 12:22 AM, Igor M igor...@gmail.com wrote: On Sun, Oct 27, 2013 at 12:17 AM, Chris Murphy li...@colorremedies.com wrote: On Oct 26, 2013, at 3:53 PM, Igor M igor...@gmail.com wrote: I even added enospc_debug mount option, still no messages. If it were kernel enospc, you should have messages in dmesg. What version of btrfs progs when making the btrfs volume? cp: failed to extend ‘/usr/local/mysql/data/gbdata/parts_0016.MYD’: No space left on device Reboot with kernel parameter ignore_loglevel and retry the copy, and see if you now have anything in dmesg at the time of the copy. Some more info. This files are MySQL database files. Tables contains only text so they compress well. But, if I copy some uncompressable file, for example some video than everything is ok, no error message, copy is successfull. I also tried mounting without compression (without compress-force=lzo) and in this case no error is reported. So it seems it's something with compression ? I'll try rebooting with ignore_loglevel parameter. It's the same error if I try to copy for ex. with midnight commander. I have the same kernel version, the same mount options, and use the same cp -a on a 5.3GB file and cannot reproduce your results. Chris Murphy Still no messages. Parameter seems to be active as /sys/module/printk/parameters/ignore_loglevel is Y, but there are no messages in log files or dmesg. Maybe I need to turn on some kernel debugging option and recompile kernel ? Also I should mention that cca 230G+ data was copied before this error started to occur. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
Didn't see before. btrfs progs were compiled today form git. # btrfs version Btrfs v0.20-rc1-358-g194aa4a -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] Btrfs-progs: allow --init-extent-tree to work when extent tree is borked
On 25/10/13 19:31, Josef Bacik wrote: On Fri, Oct 25, 2013 at 07:27:24PM +0100, Martin wrote: On 25/10/13 19:01, Josef Bacik wrote: Unfortunately you can't run --init-extent-tree if you can't actually read the extent root. Fix this by allowing partial starts with no extent root and then have fsck only check to see if the extent root is uptodate _after_ the check to see if we are init'ing the extent tree. Thanks, Signed-off-by: Josef Bacik jba...@fusionio.com --- cmds-check.c | 9 ++--- disk-io.c| 16 ++-- 2 files changed, 20 insertions(+), 5 deletions(-) diff --git a/cmds-check.c b/cmds-check.c index 69b0327..8ed7baa 100644 --- a/cmds-check.c +++ b/cmds-check.c Hey! Quick work!... Is that worth patching locally and trying against my example? Yes, I'm a little worried about your particular case so I'd like to see if it works. If you don't see a lot of output after say 5 minutes let's assume I didn't fix your problem and let me know so I can make the other change I considered. Thanks, Nope... No-go. parent transid verify failed on 911904604160 wanted 17448 found 17449 parent transid verify failed on 911904604160 wanted 17448 found 17449 parent transid verify failed on 911904604160 wanted 17448 found 17449 parent transid verify failed on 911904604160 wanted 17448 found 17449 Ignoring transid failure ...And nothing more. Looped. # gdb /sbin/btrfsck 31887 GNU gdb (Gentoo 7.5.1 p2) 7.5.1 Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type show copying and show warranty for details. This GDB was configured as x86_64-pc-linux-gnu. For bug reporting instructions, please see: http://bugs.gentoo.org/... Reading symbols from /sbin/btrfsck...Reading symbols from /usr/lib64/debug/sbin/btrfsck.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Attaching to program: /sbin/btrfsck, process 31887 warning: Could not load shared library symbols for linux-vdso.so.1. Do you need set solib-search-path or set sysroot? Reading symbols from /lib64/libuuid.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libuuid.so.1 Reading symbols from /lib64/libblkid.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libblkid.so.1 Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /usr/lib64/liblzo2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/liblzo2.so.2 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Using host libthread_db library /lib64/libthread_db.so.1. Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 0x0042b7a9 in read_extent_buffer () (gdb) (gdb) bt #0 0x0042b7a9 in read_extent_buffer () #1 0x0041ccfd in btrfs_check_node () #2 0x0041e0a2 in check_block () #3 0x0041e69e in btrfs_search_slot () #4 0x00425a6e in find_first_block_group () #5 0x00425b28 in btrfs_read_block_groups () #6 0x00421c40 in btrfs_setup_all_roots () #7 0x00421e3f in __open_ctree_fd () #8 0x00421f19 in open_ctree_fs_info () #9 0x004169b4 in cmd_check () #10 0x0040443b in main () (gdb) # btrfs version Btrfs v0.20-rc1-358-g194aa4a-dirty Emerging (1 of 1) sys-fs/btrfs-progs- Unpacking source... GIT update -- repository: git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs.git at the commit:194aa4a1bd6447bb545286d0bcb0b0be8204d79f branch: master storage directory: /usr/portage/distfiles/egit-src/btrfs-progs.git checkout type:bare repository Cloning into '/var/tmp/portage/sys-fs/btrfs-progs-/work/btrfs-progs-'... done. Branch branch-master set up to track remote branch master from origin. Switched to a new branch 'branch-master' Unpacked to /var/tmp/portage/sys-fs/btrfs-progs-/work/btrfs-progs- Source unpacked in /var/tmp/portage/sys-fs/btrfs-progs-/work Preparing source in /var/tmp/portage/sys-fs/btrfs-progs-/work/btrfs-progs- ... Source prepared. * Applying user patches from /etc/portage/patches//sys-fs/btrfs-progs- ... * jbpatch2013-10-25-extents-fix.patch ... [ ok ] * Done with patching Configuring source in /var/tmp/portage/sys-fs/btrfs-progs-/work/btrfs-progs- ... Source configured. [...] Note the compile warnings: * QA Notice: Package triggers severe warnings which
Re: No space left on device, problem
Still no messages. Parameter seems to be active as /sys/module/printk/parameters/ignore_loglevel is Y, but there are no messages in log files or dmesg. Maybe I need to turn on some kernel debugging option and recompile kernel ? Also I should mention that cca 230G+ data was copied before this error started to occur. I think I saw a similar issue before. Can you try using rsync with --bwlimit XY option to copy the files? The option will limit the speed, in kB, at which the file is being copied; it will work even when source and destination files are on a local machine. -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] Btrfs-progs: allow --init-extent-tree to work when extent tree is borked
Yup I have another plan for your situation, I will wire it up Monday and send it out. Thanks, Josef On Oct 26, 2013, at 7:16 PM, Martin m_bt...@ml1.co.uk wrote: On 25/10/13 19:31, Josef Bacik wrote: On Fri, Oct 25, 2013 at 07:27:24PM +0100, Martin wrote: On 25/10/13 19:01, Josef Bacik wrote: Unfortunately you can't run --init-extent-tree if you can't actually read the extent root. Fix this by allowing partial starts with no extent root and then have fsck only check to see if the extent root is uptodate _after_ the check to see if we are init'ing the extent tree. Thanks, Signed-off-by: Josef Bacik jba...@fusionio.com --- cmds-check.c | 9 ++--- disk-io.c| 16 ++-- 2 files changed, 20 insertions(+), 5 deletions(-) diff --git a/cmds-check.c b/cmds-check.c index 69b0327..8ed7baa 100644 --- a/cmds-check.c +++ b/cmds-check.c Hey! Quick work!... Is that worth patching locally and trying against my example? Yes, I'm a little worried about your particular case so I'd like to see if it works. If you don't see a lot of output after say 5 minutes let's assume I didn't fix your problem and let me know so I can make the other change I considered. Thanks, Nope... No-go. parent transid verify failed on 911904604160 wanted 17448 found 17449 parent transid verify failed on 911904604160 wanted 17448 found 17449 parent transid verify failed on 911904604160 wanted 17448 found 17449 parent transid verify failed on 911904604160 wanted 17448 found 17449 Ignoring transid failure ...And nothing more. Looped. # gdb /sbin/btrfsck 31887 GNU gdb (Gentoo 7.5.1 p2) 7.5.1 Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type show copying and show warranty for details. This GDB was configured as x86_64-pc-linux-gnu. For bug reporting instructions, please see: http://bugs.gentoo.org/... Reading symbols from /sbin/btrfsck...Reading symbols from /usr/lib64/debug/sbin/btrfsck.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Attaching to program: /sbin/btrfsck, process 31887 warning: Could not load shared library symbols for linux-vdso.so.1. Do you need set solib-search-path or set sysroot? Reading symbols from /lib64/libuuid.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libuuid.so.1 Reading symbols from /lib64/libblkid.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libblkid.so.1 Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /usr/lib64/liblzo2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/liblzo2.so.2 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Using host libthread_db library /lib64/libthread_db.so.1. Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 0x0042b7a9 in read_extent_buffer () (gdb) (gdb) bt #0 0x0042b7a9 in read_extent_buffer () #1 0x0041ccfd in btrfs_check_node () #2 0x0041e0a2 in check_block () #3 0x0041e69e in btrfs_search_slot () #4 0x00425a6e in find_first_block_group () #5 0x00425b28 in btrfs_read_block_groups () #6 0x00421c40 in btrfs_setup_all_roots () #7 0x00421e3f in __open_ctree_fd () #8 0x00421f19 in open_ctree_fs_info () #9 0x004169b4 in cmd_check () #10 0x0040443b in main () (gdb) # btrfs version Btrfs v0.20-rc1-358-g194aa4a-dirty Emerging (1 of 1) sys-fs/btrfs-progs- Unpacking source... GIT update -- repository: git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs.git at the commit:194aa4a1bd6447bb545286d0bcb0b0be8204d79f branch: master storage directory: /usr/portage/distfiles/egit-src/btrfs-progs.git checkout type:bare repository Cloning into '/var/tmp/portage/sys-fs/btrfs-progs-/work/btrfs-progs-'... done. Branch branch-master set up to track remote branch master from origin. Switched to a new branch 'branch-master' Unpacked to /var/tmp/portage/sys-fs/btrfs-progs-/work/btrfs-progs- Source unpacked in /var/tmp/portage/sys-fs/btrfs-progs-/work Preparing source in /var/tmp/portage/sys-fs/btrfs-progs-/work/btrfs-progs- ... Source prepared. * Applying user patches from /etc/portage/patches//sys-fs/btrfs-progs- ... *
