On 2021-01-04 21:51, cedric.dew...@eclipso.eu wrote:
I have a master NAS that makes one read only snapshot of my data per day. I want to transfer
these snapshots to a slave NAS over a slow, unreliable internet connection. (it's a cheap
provider). This rules out a "btrfs send -> ssh -> btrfs
On 05/01/2021 08:34, Forza wrote:
>
>
> On 2021-01-04 21:51, cedric.dew...@eclipso.eu wrote:
>> I have a master NAS that makes one read only snapshot of my data per
>> day. I want to transfer these snapshots to a slave NAS over a slow,
>> unreliable internet connection. (it's a cheap provider).
On Tue, Jan 5, 2021 at 5:27 AM Chris Murphy wrote:
>
> New new plan, ngompa built it for us in Fedora copr.
>
> sudo dnf install
> https://download.copr.fedorainfracloud.org/results/ngompa/btrfsprogs-robustimg/fedora-33-x86_64/01858610-btrfs-progs/btrfs-progs-5.9+git20210104.5fa05dd-0.fc33.1.x86_6
I'm keeping the rest of the mail as I received it, it's completely
garbled and would need manual reformatting. The changelog and even the
diff are completely on one line(!).
On Tue, Jan 05, 2021 at 11:08:42AM +0800, bodefang wrote:
> Similar to commmit<389305b2aa68>（“btrfs: relocation: Only remove
On Tue, 5 Jan 2021 11:24:24 +
Graham Cobb wrote:
> used that approach as the (old) NAS I was using had a very old linux
> version and didn't even run btrfs.
One anecdote --
I do use an old D-Link DNS-323 NAS with old kernel and distro (older Debian),
and only ~60 MB of RAM to serve a 8 TB d
I also thought about a different approach in the past:
1. Take a snapshot and rsync it to the server.
2. When it succeeds, make it readonly and take a note on the remote
site that indicates the Received_UUID and checksum of entire
subvolume.
3. When you want to send your diff, run `btrfs send -p .
On Mon, Jan 04, 2021 at 02:56:59PM -0800, Rosen Penev wrote:
> On Mon, Jan 4, 2021 at 6:46 AM David Sterba wrote:
> >
> > On Sat, Jan 02, 2021 at 07:50:38PM -0800, Rosen Penev wrote:
> > > I've noticed that internally, btrfs' XOR code is CPU only. Does anyone
> > > know if there is a performance a
On Fri, Dec 25, 2020 at 08:15:11PM +0100, Adam Borowski wrote:
> Signed-off-by: Adam Borowski
Thanks, added to devel.
On Fri, Dec 25, 2020 at 08:15:30PM +0100, Adam Borowski wrote:
> Any use of a long option would crash.
>
> Signed-off-by: Adam Borowski
Omar sent this fix within another series, I've added it to devel some
time ago.
https://lore.kernel.org/linux-btrfs/e0aff7fddf35f0f18ff21d1e2e50a5d127254392.16
On Fri, Dec 25, 2020 at 04:45:38PM +0900, Jiachen YANG wrote:
> Currently btrfs-convert only copies ext2 inode timestamps
> i_[cma]time from ext4, while filling 0 to nsec and crtime fields.
>
> This change copies nsec and crtime by parsing i_[cma]time_extra fields.
Thanks, added to devel, with so
>> I was expecting btrfs to do almost all reads from the fast SSD, as both
the data and the metadata is on that drive, so the slow hdd is only really
needed when there's a bitflip on the SSD, and the data has to be reconstructed.
> IIRC there will be some read policy feature to do that, but not ye
FWIW, just recompiled with the patch to be 100% sure, as I still had
the problematic FS around untouched:
[ 199.722122] BTRFS info (device dm-10): balance: start
-dvrange=34625344765952..34625344765953
[ 199.730267] BTRFS info (device dm-10): relocating block group 34625344765952
flags data|ra
January 5, 2021 7:20 PM, cedric.dew...@eclipso.eu wrote:
>>> I was expecting btrfs to do almost all reads from the fast SSD, as both
>
> the data and the metadata is on that drive, so the slow hdd is only really
> needed when there's a bitflip on the SSD, and the data has to be
> reconstructed.
On 12/12/20 5:29 PM, Matthew Wilcox wrote:
> And most urgently, when should we have the GUP meeting? On the call,
> I suggested Friday the 8th of January, but I'm happy to set something
> up for next week if we'd like to talk more urgently. Please propose a
> date & time. I know we have people i
The lowmem mode excludes all referenced blocks from the allocator in
order to avoid accidentally overwriting blocks while fixing the file
system. However for leaves it wouldn't exclude anything, it would just
pin them down, which gets cleaned up on transaction commit. We're safe
for the first mod
On Tue, Jan 5, 2021 at 7:35 AM David Sterba wrote:
>
> On Mon, Jan 04, 2021 at 02:56:59PM -0800, Rosen Penev wrote:
> > On Mon, Jan 4, 2021 at 6:46 AM David Sterba wrote:
> > >
> > > On Sat, Jan 02, 2021 at 07:50:38PM -0800, Rosen Penev wrote:
> > > > I've noticed that internally, btrfs' XOR code
PAGE_CLEAR_DIRTY and PAGE_SET_WRITEBACK are two macros used in
__process_pages_contig(), to inform the function to clear page dirty and
then set page writeback.
However page write back and dirty are two conflict status (at least for
sector size == PAGE_SIZE case), this means those two macros are a
For those functions, to support subpage size they just need to call
btrfs_page_set/clear_uptodate() wrappers.
Signed-off-by: Qu Wenruo
---
fs/btrfs/extent_io.c | 11 +++
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index 3bd4b9
This patch introduce the following functions to handle btrfs subpage
error status:
- btrfs_subpage_set_error()
- btrfs_subpage_clear_error()
- btrfs_subpage_test_error()
Those helpers can only be called when the range is ensured to be
inside the page.
- btrfs_page_set_error()
- btrfs_page_clea
For subpage case, grab_extent_buffer() can't really get an extent buffer
just from btrfs_subpage.
Although we have btrfs_subpage::tree_block_bitmap, which can be used to
grab the bytenr of an existing extent buffer, and can then go radix tree
search to grab that existing eb.
However we are still
Unlike the original try_release_extent_buffer,
try_release_subpage_extent_buffer() will iterate through
btrfs_subpage::tree_block_bitmap, and try to release each extent buffer.
Signed-off-by: Qu Wenruo
---
fs/btrfs/extent_io.c | 76
1 file changed, 76
The refactor involves the following modifications:
- iosize alignment
In fact we don't really need to manually do alignment at all.
All extent maps should already be aligned, thus basic ASSERT() check
would be enough.
- redundant variables
We have extra variable like blocksize/pg_offset/en
For subpage metadata validation check, there are some difference:
- Read must finish in one bvec
Since we're just reading one subpage range in one page, it should
never be split into two bios nor two bvecs.
- How to grab the existing eb
Instead of grabbing eb using page->private, we have to
To handle subpage status update, add the following new tricks:
- Use btrfs_page_*() helpers to update page status
Now we can handle both cases well.
- No page unlock for subpage metadata
Since subpage metadata doesn't utilize page locking at all, skip it.
For subpage data locking, it's handl
When __process_pages_contig() get called for
extent_clear_unlock_delalloc(), if we hit the locked page, only Private2
bit is updated, but dirty/writeback/error bits are all skiped.
There are several call sites call extent_clear_unlock_delalloc() with
@locked_page and PAGE_CLEAR_DIRTY/PAGE_SET_WRIT
Even for regular btrfs, there are locations where we allocate dummy
extent buffers for temporary usage.
Like tree_mod_log_rewind() and get_old_root().
Those dummy extent buffers will be handled by the same eb accessors, and
if they don't have page::private subpage eb accessors can fail.
To addre
In btrfs_release_extent_buffer_pages(), we need to add extra handling
for subpage.
To do so, introduce a new helper, detach_extent_buffer_page(), to do
different handling for regular and subpage cases.
For subpage case, the new trick is to clear the range of current extent
buffer, and detach page
This patch will extract the code to grab an extent buffer from a page
into a helper, grab_extent_buffer_from_page().
This reduces one indent level, and provides the work place for later
expansion for subapge support.
Reviewed-by: Johannes Thumshirn
Signed-off-by: Qu Wenruo
---
fs/btrfs/extent_
This patch introduce the following functions to handle btrfs subpage
uptodate status:
- btrfs_subpage_set_uptodate()
- btrfs_subpage_clear_uptodate()
- btrfs_subpage_test_uptodate()
Those helpers can only be called when the range is ensured to be
inside the page.
- btrfs_page_set_uptodate()
-
Patches can be fetched from github:
https://github.com/adam900710/linux/tree/subpage
Currently the branch also contains partial RW data support (still some
out-of-sync subpage data page status).
Great thanks to David for his effort reviewing and merging the
preparation patches into misc-next.
Now
The parameter @offset can't be more confusing.
In fact that parameter is the disk bytenr for metadata/data.
Rename it to @disk_bytenr and update the comment to reduce confusion.
Since we're here, also rename all @offset passed into
submit_extent_page() to @disk_bytenr.
Reviewed-by: Josef Bacik
For btrfs subpage support, we need a structure to record extra info for
the status of each sectors of a page.
This patch will introduce the skeleton structure for future btrfs
subpage support.
All subpage related code would go to subpage.[ch] to avoid populating
the existing code base.
Reviewed-b
For subpage case, we need to allocate new memory for each metadata page.
So we need to:
- Allow attach_extent_buffer_page() to return int
To indicate allocation failure
- Prealloc page->private for alloc_extent_buffer()
We don't want to call memory allocation with spinlock hold, so
do preal
To support subpage sector size, data also need extra info to make sure
which sectors in a page are uptodate/dirty/...
This patch will make pages for data inodes to get btrfs_subpage
structure attached, and detached when the page is freed.
This patch also slightly changes the timing when
set_page_
The original comment is from the initial merge, which has several
problems:
- No holes check any more
- No inline decision is made
Update the out-of-date comment with more correct one.
Signed-off-by: Qu Wenruo
---
fs/btrfs/file.c | 11 +--
1 file changed, 5 insertions(+), 6 deletions(-)
In btrfs data page read path, the page status update are handled in two
different locations:
btrfs_do_read_page()
{
while (cur <= end) {
/* No need to read from disk */
if (HOLE/PREALLOC/INLINE){
memset();
This adds the basic RO mount ability for 4K sector size on 64K page
system.
Currently we only plan to support 4K and 64K page system.
Signed-off-by: Qu Wenruo
---
fs/btrfs/disk-io.c | 24 +---
fs/btrfs/super.c | 7 +++
2 files changed, 28 insertions(+), 3 deletions(-)
For btrfs_clone_extent_buffer(), it's mostly the same code of
__alloc_dummy_extent_buffer(), except it has extra page copy.
So to make it subpage compatible, we only need to:
- Call set_extent_buffer_uptodate() instead of SetPageUptodate()
This will set correct uptodate bit for subpage and regul
Introduce a new helper, read_extent_buffer_subpage(), to do the subpage
extent buffer read.
The difference between regular and subpage routines are:
- No page locking
Here we completely rely on extent locking.
Page locking can reduce the concurrency greatly, as if we lock one
page to read on
On Wed 06 Jan 2021 at 05:40, Josef Bacik
wrote:
The lowmem mode excludes all referenced blocks from the
allocator in
order to avoid accidentally overwriting blocks while fixing the
file
system. However for leaves it wouldn't exclude anything, it
would just
pin them down, which gets clean
I have put a SSD and a slow laptop HDD in btrfs raid. This was a bad
idea, my system does not feel responsive. When i load a program, dstat
shows half of the program is loaded from the SSD, and the rest from
the slow hard drive.
The drive speeds are evolving. NVME introduced lower latency
Hi Qu,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on kdave/for-next]
[also build test WARNING on v5.11-rc2 next-20210104]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documente
On 2021/1/6 下午1:04, kernel test robot wrote:
Hi Qu,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on kdave/for-next]
[also build test WARNING on v5.11-rc2 next-20210104]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitt
On 1/6/21 1:32 PM, Qu Wenruo wrote:
On 2021/1/6 下午1:04, kernel test robot wrote:
Hi Qu,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on kdave/for-next]
[also build test WARNING on v5.11-rc2 next-20210104]
[If your patch is applied to the wrong git tree,
On 2021/1/6 上午9:01, Qu Wenruo wrote:
For subpage case, we need to allocate new memory for each metadata page.
So we need to:
- Allow attach_extent_buffer_page() to return int
To indicate allocation failure
- Prealloc page->private for alloc_extent_buffer()
We don't want to call memory
In the meantime, since I have sent the base patch as below [1], the
block layer commit 0d02129e76ed (block: merge struct block_device and
struct hd_struct) has changed the first argument in the function
part_stat_read_all() to struct block_device in 5.11-rc1. So the
compilation will fail. This patc
46 matches
Mail list logo
