send | receive: received snapshot is missing recent files

2017-09-05 Thread Dave
I'm running Arch Linux on BTRFS. I use Snapper to take hourly snapshots and it works without any issues. I have a bash script that uses send | receive to transfer snapshots to a couple external HDD's. The script runs daily on a systemd timer. I set all this up recently and I first noticed that it

Re: [PATCH 0/5] Mkfs: Rework --rootdir to a more generic behavior

2017-09-05 Thread Qu Wenruo
On 2017年09月06日 03:05, Goffredo Baroncelli wrote: On 09/05/2017 10:19 AM, Qu Wenruo wrote: On 2017年09月05日 02:08, David Sterba wrote: On Mon, Sep 04, 2017 at 03:41:05PM +0900, Qu Wenruo wrote: mkfs.btrfs --rootdir provides user a method to generate btrfs with pre-written content while withou

Re: \o/ compsize

2017-09-05 Thread Adam Borowski
On Mon, Sep 04, 2017 at 10:33:40PM +0200, A L wrote: > On 9/4/2017 5:11 PM, Adam Borowski wrote: > > Hi! > > Here's an utility to measure used compression type + ratio on a set of files > > or directories: https://github.com/kilobyte/compsize > > Great tool. Just tried it on some of my backup snap

[PATCH 1/2] btrfs-progs: Use clean_tree_block when btrfs_update_root fail

2017-09-05 Thread Gu Jinxiang
In btrfs_fsck_reinit_root, when btrfs_alloc_free_block fail, it will update on original root. Before update it, used btrfs_mark_buffer_dirty to set the flag to EXTENT_DIRTY. So, we should call clean_tree_block to clear the flag if update fail. Signed-off-by: Gu Jinxiang --- cmds-check.c | 1 + 1

[PATCH 2/2] btrfs-progs: Replace some BUG_ON by return

2017-09-05 Thread Gu Jinxiang
The following test failed becasuse there is no data/metadata/system block_group. Use return ret to replace BUG_ON(ret) to avoid system crash, because there is enough message for user to understand what happened. $sudo TEST=003\* make test-fuzz Unable to find block group for 0 Unable to find block

Re: read-only for no good reason on 4.9.30

2017-09-05 Thread Chris Murphy
On Sun, Sep 3, 2017 at 11:19 PM, Russell Coker wrote: > I have a system with less than 50% disk space used. It just started rejecting > writes due to lack of disk space. What's the error? Is it ENOSPC? Kinda needs kernel messages, and also if it's ENOSPC to have mounted with enospc_debug option.

Re: [PATCH 0/5] Mkfs: Rework --rootdir to a more generic behavior

2017-09-05 Thread Goffredo Baroncelli
On 09/05/2017 10:19 AM, Qu Wenruo wrote: > > > On 2017年09月05日 02:08, David Sterba wrote: >> On Mon, Sep 04, 2017 at 03:41:05PM +0900, Qu Wenruo wrote: >>> mkfs.btrfs --rootdir provides user a method to generate btrfs with >>> pre-written content while without the need of root privilege. >>> >>> H

Re: \o/ compsize

2017-09-05 Thread Nick Terrell
On 9/4/17, 8:12 AM, "Adam Borowski" wrote: > Hi! > Here's an utility to measure used compression type + ratio on a set of files > or directories: https://github.com/kilobyte/compsize > > It should be of great help for users, and also if you: > * muck with compression levels > * add new compressio

Re: BTRFS: error (device dm-2) in btrfs_run_delayed_refs:2960: errno=-17 Object already exists (since 3.4 / 2012)

2017-09-05 Thread Josef Bacik
Alright I just reworked the build tree ref stuff and tested it to make sure it wasn’t going to give false positives again. Apparently I had only ever used this with very basic existing fs’es and nothing super complicated, so it was just broken for anything complex. I’ve pushed it to my tree, y

Re: Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]

2017-09-05 Thread Liu Bo
On Tue, Sep 05, 2017 at 11:47:26AM +0200, Marco Lorenzo Crociani wrote: > Hi, > I was transferring some data with rsync to a btrfs filesystem when I got: > > set 04 14:59:05 kernel: INFO: task kworker/u33:2:25015 blocked for more > than 120 seconds. > set 04 14:59:05 kernel: Not tainted 4.

Re: btrfs check --repair now runs in minutes instead of hours? aborting

2017-09-05 Thread Marc MERLIN
On Tue, Sep 05, 2017 at 04:05:04PM +0800, Qu Wenruo wrote: > > gargamel:~# btrfs fi df /mnt/btrfs_pool1 > > Data, single: total.60TiB, used.54TiB > > System, DUP: total2.00MiB, used=1.19MiB > > Metadata, DUP: totalX.00GiB, used.69GiB > > Wait for a minute. > > Is that .69GiB means 706 MiB? Or

Re: Is autodefrag recommended? -- re-duplication???

2017-09-05 Thread Hugo Mills
On Tue, Sep 05, 2017 at 05:01:10PM +0300, Marat Khalili wrote: > Dear experts, > > At first reaction to just switching autodefrag on was positive, but > mentions of re-duplication are very scary. Main use of BTRFS here is > backup snapshots, so re-duplication would be disastrous. > > In order to

Re: \o/ compsize

2017-09-05 Thread Qu Wenruo
On 2017年09月05日 22:21, Hans van Kranenburg wrote: On 09/05/2017 04:02 PM, Qu Wenruo wrote: On 2017年09月05日 03:52, Timofey Titovets wrote: 2017-09-04 21:42 GMT+03:00 Adam Borowski : On Mon, Sep 04, 2017 at 07:07:25PM +0300, Timofey Titovets wrote: 2017-09-04 18:11 GMT+03:00 Adam Borowski :

Re: \o/ compsize

2017-09-05 Thread Hans van Kranenburg
On 09/05/2017 04:02 PM, Qu Wenruo wrote: > > > On 2017年09月05日 03:52, Timofey Titovets wrote: >> 2017-09-04 21:42 GMT+03:00 Adam Borowski : >>> On Mon, Sep 04, 2017 at 07:07:25PM +0300, Timofey Titovets wrote: 2017-09-04 18:11 GMT+03:00 Adam Borowski : > Here's an utility to measure used

Re: \o/ compsize

2017-09-05 Thread Qu Wenruo
On 2017年09月05日 03:52, Timofey Titovets wrote: 2017-09-04 21:42 GMT+03:00 Adam Borowski : On Mon, Sep 04, 2017 at 07:07:25PM +0300, Timofey Titovets wrote: 2017-09-04 18:11 GMT+03:00 Adam Borowski : Here's an utility to measure used compression type + ratio on a set of files or directories: h

Re: Is autodefrag recommended? -- re-duplication???

2017-09-05 Thread Marat Khalili
Dear experts, At first reaction to just switching autodefrag on was positive, but mentions of re-duplication are very scary. Main use of BTRFS here is backup snapshots, so re-duplication would be disastrous. In order to stick to concrete example, let there be two files, 4KB and 4GB in size,

Re: Is autodefrag recommended?

2017-09-05 Thread Austin S. Hemmelgarn
On 2017-09-05 08:49, Henk Slager wrote: On Tue, Sep 5, 2017 at 1:45 PM, Austin S. Hemmelgarn wrote: - You end up duplicating more data than is strictly necessary. This is, IIRC, something like 128 KiB for a write. FWIW< I'm pretty sure you can mitigate this first issue by running a r

Re: How to disable/revoke 'compression'?

2017-09-05 Thread Qu Wenruo
On 2017年09月05日 19:36, Austin S. Hemmelgarn wrote: On 2017-09-03 19:55, Qu Wenruo wrote: On 2017年09月04日 02:06, Adam Borowski wrote: On Sun, Sep 03, 2017 at 07:32:01PM +0200, Cloud Admin wrote: Hi, I used the mount option 'compression' on some mounted sub volumes. How can I revoke the compre

Re: Is autodefrag recommended?

2017-09-05 Thread Henk Slager
On Tue, Sep 5, 2017 at 1:45 PM, Austin S. Hemmelgarn wrote: >> - You end up duplicating more data than is strictly necessary. This >> is, IIRC, something like 128 KiB for a write. > > FWIW< I'm pretty sure you can mitigate this first issue by running a regular > defrag on a semi-regular bas

Re: Is autodefrag recommended?

2017-09-05 Thread A L
There is a drawback in that defragmentation re-dups data that is previously deduped or shared in snapshots/subvolumes. From: Marat Khalili -- Sent: 2017-09-04 - 11:31 > Hello list, > good time of the day, > > More than once I see mentioned in this list that autodefrag option > solve

Re: Is autodefrag recommended?

2017-09-05 Thread Austin S. Hemmelgarn
On 2017-09-04 06:54, Hugo Mills wrote: On Mon, Sep 04, 2017 at 12:31:54PM +0300, Marat Khalili wrote: Hello list, good time of the day, More than once I see mentioned in this list that autodefrag option solves problems with no apparent drawbacks, but it's not the default. Can you recommend to j

Re: How to disable/revoke 'compression'?

2017-09-05 Thread Austin S. Hemmelgarn
On 2017-09-03 19:55, Qu Wenruo wrote: On 2017年09月04日 02:06, Adam Borowski wrote: On Sun, Sep 03, 2017 at 07:32:01PM +0200, Cloud Admin wrote: Hi, I used the mount option 'compression' on some mounted sub volumes. How can I revoke the compression? Means to delete the option and get all data un

Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]

2017-09-05 Thread Marco Lorenzo Crociani
Hi, I was transferring some data with rsync to a btrfs filesystem when I got: set 04 14:59:05 kernel: INFO: task kworker/u33:2:25015 blocked for more than 120 seconds. set 04 14:59:05 kernel: Not tainted 4.12.10-1.el7.elrepo.x86_64 #1 set 04 14:59:05 kernel: "echo 0 > /proc/sys/kernel

Re: btrfs check --repair now runs in minutes instead of hours? aborting

2017-09-05 Thread Duncan
Qu Wenruo posted on Tue, 05 Sep 2017 17:06:35 +0800 as excerpted: >> See if these numbers, copied and reformatted from his post with spaces >> inserted either side of the numbers and the equals signs deleted, >> arrive any less garbled: >> >> Data, single: total 10.60 TiB, used 10.54 TiB System,

Re: btrfs check --repair now runs in minutes instead of hours? aborting

2017-09-05 Thread Qu Wenruo
On 2017年09月05日 16:54, Duncan wrote: Qu Wenruo posted on Tue, 05 Sep 2017 16:05:04 +0800 as excerpted: On 2017年09月05日 10:55, Marc MERLIN wrote: gargamel:~# btrfs fi df /mnt/btrfs_pool1 Data, single: total.60TiB, used.54TiB System, DUP: total2.00MiB, used=1.19MiB Metadata, DUP: totalX.00GiB

Re: btrfs check --repair now runs in minutes instead of hours? aborting

2017-09-05 Thread Duncan
Qu Wenruo posted on Tue, 05 Sep 2017 16:05:04 +0800 as excerpted: > On 2017年09月05日 10:55, Marc MERLIN wrote: >> >> gargamel:~# btrfs fi df /mnt/btrfs_pool1 >> Data, single: total.60TiB, used.54TiB >> System, DUP: total2.00MiB, used=1.19MiB >> Metadata, DUP: totalX.00GiB, used.69GiB > > Wait f

Re: [PATCH 0/5] Mkfs: Rework --rootdir to a more generic behavior

2017-09-05 Thread Qu Wenruo
On 2017年09月05日 02:08, David Sterba wrote: On Mon, Sep 04, 2017 at 03:41:05PM +0900, Qu Wenruo wrote: mkfs.btrfs --rootdir provides user a method to generate btrfs with pre-written content while without the need of root privilege. However the code is quite old and doesn't get much review or te

Re: btrfs check --repair now runs in minutes instead of hours? aborting

2017-09-05 Thread Qu Wenruo
On 2017年09月05日 10:55, Marc MERLIN wrote: On Tue, Sep 05, 2017 at 09:21:55AM +0800, Qu Wenruo wrote: On 2017年09月05日 09:05, Marc MERLIN wrote: Ok, I don't want to sound like I'm complaining :) but I updated btrfs-progs to top of tree in git, installed it, and ran it on an 8TiB filesystem that