Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-24 Thread Dave Chinner
On Mon, Jun 24, 2013 at 10:37:57AM +, Sidorov, Andrei wrote:
> On 24.06.2013 13:36, Namjae Jeon wrote:
> > Currently, we can try implementing dave's suggesstion of introducing a
> > new flag FALLOC_FL_COLLAPSE_RANGE for falloctae instead of individual
> > ioctls for both XFS and EXT4. Thanks.
> 
> Hi,
> 
> Currently PUNCH_HOLE requires KEEP_SIZE to be set as well. I think there
> is no need to invent COLLAPSE_RANGE, but instead fallocate should
> support PUNCH_HOLE without KEEP_SIZE. However I'm not sure that putting
> block alignment restriction is a right way to go.

No, it doesn't make sense. Punching a hole *leaves a hole* in the
file, not that all the data on the high side of the region being
punched out is *moved* into the hole that was punched out. i.e.
PUNCH_HOLE does not affect data outside the range of the holebeing
punched, while COLLAPSE_RANGE significantly affects data in the
file outside the range being removed. Hence that are not the same
operation at all.

FYI, what KEEP_SIZE means for punching a hole is that preserve the
file size is explicitly required by the operation. It's
documentation of the expected behaviour more than anything.  i.e if
the hole is at the end of the file, punching a hole won't -truncate-
the file.  Also, you can punch holes *beyond EOF* (so you can punch
blocks allocated beyond EOF) and KEEP_SIZE means that it doesn't
change the file size when it does this.

Cheers,

Dave.
-- 
Dave Chinner
da...@fromorbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-24 Thread Zheng Liu
On Mon, Jun 24, 2013 at 10:37:57AM +, Sidorov, Andrei wrote:
> On 24.06.2013 13:36, Namjae Jeon wrote:
> > Currently, we can try implementing dave's suggesstion of introducing a
> > new flag FALLOC_FL_COLLAPSE_RANGE for falloctae instead of individual
> > ioctls for both XFS and EXT4. Thanks.
> 
> Hi,
> 
> Currently PUNCH_HOLE requires KEEP_SIZE to be set as well. I think there
> is no need to invent COLLAPSE_RANGE, but instead fallocate should
> support PUNCH_HOLE without KEEP_SIZE. However I'm not sure that putting
> block alignment restriction is a right way to go.

PUNCH_HOLE without KEEP_SIZE makes sense to me.

Regards,
- Zheng
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-24 Thread Sidorov, Andrei
On 24.06.2013 13:36, Namjae Jeon wrote:
> Currently, we can try implementing dave's suggesstion of introducing a
> new flag FALLOC_FL_COLLAPSE_RANGE for falloctae instead of individual
> ioctls for both XFS and EXT4. Thanks.

Hi,

Currently PUNCH_HOLE requires KEEP_SIZE to be set as well. I think there
is no need to invent COLLAPSE_RANGE, but instead fallocate should
support PUNCH_HOLE without KEEP_SIZE. However I'm not sure that putting
block alignment restriction is a right way to go.

Regards,
Andrey.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-24 Thread Namjae Jeon
2013/6/24, Christoph Hellwig :
> On Mon, Jun 24, 2013 at 12:44:59PM +1000, Dave Chinner wrote:
>> various DVR companies shipping equivalent ioctl-based functionality
>> for XFS filesystems to both insert and remove chunks in video
>> streams, but I've never been able to find the code for it anywhere.
>
Hi. Christoph.
> Samsung does this, also for UDF.  Namjae, could you please submit these
> two as well, including a proper VFS interface?
Yes, Sure.
Intially, we also thought of adding a new flag to fallocate instead of ioctls.
But as Ted rightly pointed, fallocate works on byte granularity and we
have block aligned restrictions, so once we decided to go for ioctls
and hear other people opninons.
Currently, we can try implementing dave's suggesstion of introducing a
new flag FALLOC_FL_COLLAPSE_RANGE  for falloctae instead of individual
ioctls for both XFS and EXT4.

Thanks.
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-24 Thread Dave Chinner
On Sun, Jun 23, 2013 at 11:12:35PM -0400, Theodore Ts'o wrote:
> On Mon, Jun 24, 2013 at 12:44:59PM +1000, Dave Chinner wrote:
> > 
> > Hence, at minimum, this should be a fallocate() operation, not a ext4
> > specific ioctl as it is relatively trivial to implement on most
> > extent based filesystems.
> 
> The fallocate() uses a units of bytes for the offset and length; would
> a FALLOC_FL_COLLAPSE_RANGE be guaranteed to work on any arbitrary
> offset and length?  Or would it only work if the offset and length are
> multiples of the file system blocksize?

There's nothing stopping us from restricting the offset/len to
specific alignments if the operation cannot be done on arbitrary
byte ranges. We do that for direct IO, and the sky hasn't fallen
yet.

> The the EXT4_IOC_TRUNCATE_BLOCK_RANGE interface solves this problem by
> using units of file system blocks (i.e., __u32 start_block), but that
> raises another issue, which is it forces the user space program to
> somehow figure out the file system block size, which seems a bit nasty.

Yeah, exactly. We can do that internally very easily, and EINVAL can
be returned when the alignment is bad just like we do for direct
IO...

But, well, I pine for a generic XFS_IOC_DIOINFO interface so the
filesystem can tell users about alignment restrictions

Cheers,

Dave.
-- 
Dave Chinner
da...@fromorbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-24 Thread Christoph Hellwig
On Mon, Jun 24, 2013 at 12:44:59PM +1000, Dave Chinner wrote:
> various DVR companies shipping equivalent ioctl-based functionality
> for XFS filesystems to both insert and remove chunks in video
> streams, but I've never been able to find the code for it anywhere.

Samsung does this, also for UDF.  Namjae, could you please submit these
two as well, including a proper VFS interface?

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-24 Thread Namjae Jeon
2013/6/24, Andreas Dilger :
> On 2013-06-23, at 0:07, Namjae Jeon  wrote:
>
>> From: Namjae Jeon 
>>
>> This patch series introduces 2 new ioctls for ext4.
>>
>> Truncate_block_range ioctl truncates blocks from source file.
>
Hi. Andreas.
> How is this different from fallocate(FALLOC_FL_PUNCH_HOLE)?  That is already
> in existing kernels, and portable across multiple filesystems.
Dave chinner already answered it.

>
>> Transfer_block_range ioctl transfers data blocks from source file
>> and append them at the end of destination file.
>
> There is already a similar ioctl for defragmenting files. Is it possible to
> use that, or does it have different semantics?

For defrag ioctl:
struct move_extent {
__u32 reserved; /* should be zero */
__u32 donor_fd; /* donor file descriptor */
__u64 orig_start;   /* logical start offset in block for orig */
__u64 donor_start;  /* logical start offset in block for donor */
__u64 len;  /* block length to be moved */
__u64 moved_len;/* moved block length */
};

For this ioctl to work, there are some pre conditions:
1) Contiguous blocks for donor_fd are allocated in user space using fallocate.
2) orig_start and donor_start should be same.

The fundamental difference between these 2 ioctls is that in defrag
ioctl data is copied, (page by page?) from the extents of source file
to the newly allocated extents of donor file. However, in transfer
block range ioctl the extents are transfered from the source file and
appended at the end of donor file. There is only metadata movement.

Thanks.
>
>> Ioctl1:EXT4_IOC_TRUNCATE_BLOCK_RANGE:
>> This ioctl truncates a range of data blocks from file.
>> It is useful to remove easily and quickly the garbage data
>> at the middle of file.
>>
>> e.g. we have a movie file and there is long advertisement in movie file.
>> user want to remove only advertisement range.
>
> While this works in theory, there is very little chance that the movie data
> will align exactly to filesystem block boundaries.
>
> Cheers, Andreas
>
>> 1) Movie file (8GB), There is the adverisement of 500MB size.
>> 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-24 Thread Namjae Jeon
2013/6/24, Andreas Dilger adil...@dilger.ca:
 On 2013-06-23, at 0:07, Namjae Jeon linkinj...@gmail.com wrote:

 From: Namjae Jeon namjae.j...@samsung.com

 This patch series introduces 2 new ioctls for ext4.

 Truncate_block_range ioctl truncates blocks from source file.

Hi. Andreas.
 How is this different from fallocate(FALLOC_FL_PUNCH_HOLE)?  That is already
 in existing kernels, and portable across multiple filesystems.
Dave chinner already answered it.


 Transfer_block_range ioctl transfers data blocks from source file
 and append them at the end of destination file.

 There is already a similar ioctl for defragmenting files. Is it possible to
 use that, or does it have different semantics?

For defrag ioctl:
struct move_extent {
__u32 reserved; /* should be zero */
__u32 donor_fd; /* donor file descriptor */
__u64 orig_start;   /* logical start offset in block for orig */
__u64 donor_start;  /* logical start offset in block for donor */
__u64 len;  /* block length to be moved */
__u64 moved_len;/* moved block length */
};

For this ioctl to work, there are some pre conditions:
1) Contiguous blocks for donor_fd are allocated in user space using fallocate.
2) orig_start and donor_start should be same.

The fundamental difference between these 2 ioctls is that in defrag
ioctl data is copied, (page by page?) from the extents of source file
to the newly allocated extents of donor file. However, in transfer
block range ioctl the extents are transfered from the source file and
appended at the end of donor file. There is only metadata movement.

Thanks.

 Ioctl1:EXT4_IOC_TRUNCATE_BLOCK_RANGE:
 This ioctl truncates a range of data blocks from file.
 It is useful to remove easily and quickly the garbage data
 at the middle of file.

 e.g. we have a movie file and there is long advertisement in movie file.
 user want to remove only advertisement range.

 While this works in theory, there is very little chance that the movie data
 will align exactly to filesystem block boundaries.

 Cheers, Andreas

 1) Movie file (8GB), There is the adverisement of 500MB size.
 
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-24 Thread Christoph Hellwig
On Mon, Jun 24, 2013 at 12:44:59PM +1000, Dave Chinner wrote:
 various DVR companies shipping equivalent ioctl-based functionality
 for XFS filesystems to both insert and remove chunks in video
 streams, but I've never been able to find the code for it anywhere.

Samsung does this, also for UDF.  Namjae, could you please submit these
two as well, including a proper VFS interface?

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-24 Thread Dave Chinner
On Sun, Jun 23, 2013 at 11:12:35PM -0400, Theodore Ts'o wrote:
 On Mon, Jun 24, 2013 at 12:44:59PM +1000, Dave Chinner wrote:
  
  Hence, at minimum, this should be a fallocate() operation, not a ext4
  specific ioctl as it is relatively trivial to implement on most
  extent based filesystems.
 
 The fallocate() uses a units of bytes for the offset and length; would
 a FALLOC_FL_COLLAPSE_RANGE be guaranteed to work on any arbitrary
 offset and length?  Or would it only work if the offset and length are
 multiples of the file system blocksize?

There's nothing stopping us from restricting the offset/len to
specific alignments if the operation cannot be done on arbitrary
byte ranges. We do that for direct IO, and the sky hasn't fallen
yet.

 The the EXT4_IOC_TRUNCATE_BLOCK_RANGE interface solves this problem by
 using units of file system blocks (i.e., __u32 start_block), but that
 raises another issue, which is it forces the user space program to
 somehow figure out the file system block size, which seems a bit nasty.

Yeah, exactly. We can do that internally very easily, and EINVAL can
be returned when the alignment is bad just like we do for direct
IO...

But, well, I pine for a generic XFS_IOC_DIOINFO interface so the
filesystem can tell users about alignment restrictions

Cheers,

Dave.
-- 
Dave Chinner
da...@fromorbit.com
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-24 Thread Namjae Jeon
2013/6/24, Christoph Hellwig h...@infradead.org:
 On Mon, Jun 24, 2013 at 12:44:59PM +1000, Dave Chinner wrote:
 various DVR companies shipping equivalent ioctl-based functionality
 for XFS filesystems to both insert and remove chunks in video
 streams, but I've never been able to find the code for it anywhere.

Hi. Christoph.
 Samsung does this, also for UDF.  Namjae, could you please submit these
 two as well, including a proper VFS interface?
Yes, Sure.
Intially, we also thought of adding a new flag to fallocate instead of ioctls.
But as Ted rightly pointed, fallocate works on byte granularity and we
have block aligned restrictions, so once we decided to go for ioctls
and hear other people opninons.
Currently, we can try implementing dave's suggesstion of introducing a
new flag FALLOC_FL_COLLAPSE_RANGE  for falloctae instead of individual
ioctls for both XFS and EXT4.

Thanks.


--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-24 Thread Sidorov, Andrei
On 24.06.2013 13:36, Namjae Jeon wrote:
 Currently, we can try implementing dave's suggesstion of introducing a
 new flag FALLOC_FL_COLLAPSE_RANGE for falloctae instead of individual
 ioctls for both XFS and EXT4. Thanks.

Hi,

Currently PUNCH_HOLE requires KEEP_SIZE to be set as well. I think there
is no need to invent COLLAPSE_RANGE, but instead fallocate should
support PUNCH_HOLE without KEEP_SIZE. However I'm not sure that putting
block alignment restriction is a right way to go.

Regards,
Andrey.
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-24 Thread Zheng Liu
On Mon, Jun 24, 2013 at 10:37:57AM +, Sidorov, Andrei wrote:
 On 24.06.2013 13:36, Namjae Jeon wrote:
  Currently, we can try implementing dave's suggesstion of introducing a
  new flag FALLOC_FL_COLLAPSE_RANGE for falloctae instead of individual
  ioctls for both XFS and EXT4. Thanks.
 
 Hi,
 
 Currently PUNCH_HOLE requires KEEP_SIZE to be set as well. I think there
 is no need to invent COLLAPSE_RANGE, but instead fallocate should
 support PUNCH_HOLE without KEEP_SIZE. However I'm not sure that putting
 block alignment restriction is a right way to go.

PUNCH_HOLE without KEEP_SIZE makes sense to me.

Regards,
- Zheng
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-24 Thread Dave Chinner
On Mon, Jun 24, 2013 at 10:37:57AM +, Sidorov, Andrei wrote:
 On 24.06.2013 13:36, Namjae Jeon wrote:
  Currently, we can try implementing dave's suggesstion of introducing a
  new flag FALLOC_FL_COLLAPSE_RANGE for falloctae instead of individual
  ioctls for both XFS and EXT4. Thanks.
 
 Hi,
 
 Currently PUNCH_HOLE requires KEEP_SIZE to be set as well. I think there
 is no need to invent COLLAPSE_RANGE, but instead fallocate should
 support PUNCH_HOLE without KEEP_SIZE. However I'm not sure that putting
 block alignment restriction is a right way to go.

No, it doesn't make sense. Punching a hole *leaves a hole* in the
file, not that all the data on the high side of the region being
punched out is *moved* into the hole that was punched out. i.e.
PUNCH_HOLE does not affect data outside the range of the holebeing
punched, while COLLAPSE_RANGE significantly affects data in the
file outside the range being removed. Hence that are not the same
operation at all.

FYI, what KEEP_SIZE means for punching a hole is that preserve the
file size is explicitly required by the operation. It's
documentation of the expected behaviour more than anything.  i.e if
the hole is at the end of the file, punching a hole won't -truncate-
the file.  Also, you can punch holes *beyond EOF* (so you can punch
blocks allocated beyond EOF) and KEEP_SIZE means that it doesn't
change the file size when it does this.

Cheers,

Dave.
-- 
Dave Chinner
da...@fromorbit.com
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-23 Thread Theodore Ts'o
On Mon, Jun 24, 2013 at 12:44:59PM +1000, Dave Chinner wrote:
> 
> Hence, at minimum, this should be a fallocate() operation, not a ext4
> specific ioctl as it is relatively trivial to implement on most
> extent based filesystems.

The fallocate() uses a units of bytes for the offset and length; would
a FALLOC_FL_COLLAPSE_RANGE be guaranteed to work on any arbitrary
offset and length?  Or would it only work if the offset and length are
multiples of the file system blocksize?

The the EXT4_IOC_TRUNCATE_BLOCK_RANGE interface solves this problem by
using units of file system blocks (i.e., __u32 start_block), but that
raises another issue, which is it forces the user space program to
somehow figure out the file system block size, which seems a bit nasty.

   - Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-23 Thread Dave Chinner
On Sun, Jun 23, 2013 at 08:32:32PM -0400, Eric Sandeen wrote:
> On Jun 23, 2013, at 12:01 PM, Andreas Dilger  wrote:
> 
> > On 2013-06-23, at 0:07, Namjae Jeon  wrote:
> > 
> >> From: Namjae Jeon 
> >> 
> >> This patch series introduces 2 new ioctls for ext4.
> >> 
> >> Truncate_block_range ioctl truncates blocks from source file.
> > 
> > How is this different from fallocate(FALLOC_FL_PUNCH_HOLE)?  That is 
> > already in existing kernels, and portable across multiple filesystems. 
> > 
> Same question.  Punch hole should do this already...

Hole punch doesn't change the offsets of subsequent extents in the
file - it leaves a hole. This doesn't leave a hole at all - all the 
extents above the range are shifted down to offset where the extents
being punched out started.

> >> Transfer_block_range ioctl transfers data blocks from source file
> >> and append them at the end of destination file.
> > 
> > There is already a similar ioctl for defragmenting files. Is it possible to 
> > use that, or does it have different semantics?
> > 
> >> Ioctl1:EXT4_IOC_TRUNCATE_BLOCK_RANGE:
> >> This ioctl truncates a range of data blocks from file.
> >> It is useful to remove easily and quickly the garbage data
> >> at the middle of file.
> >> 
> >> e.g. we have a movie file and there is long advertisement in movie file.
> >> user want to remove only advertisement range.
> > 
> > While this works in theory, there is very little chance that the movie data 
> > will align exactly to filesystem block boundaries. 
> > 
> Or more importantly on compression codec boundaries.   Wouldn't this look 
> like corruption at playback time?

Not necessarily. Video codecs are encapsulated in a container that
can be used to link key frames together so you can do this sort of
manipulation of the file contents and just change an
offset-to-next-keyframe value in the container and it all just
works. Non linear editting (NLE) software has been doing this
manually for 15 years by copying data around - this just optimises
the operation by manipulating the extent mapping rather than needing
to physically copy the data.

FWIW, I've heard persistent rumors going back several years of
various DVR companies shipping equivalent ioctl-based functionality
for XFS filesystems to both insert and remove chunks in video
streams, but I've never been able to find the code for it anywhere.

Hence, at minimum, this should be a fallocate() operation, not a ext4
specific ioctl as it is relatively trivial to implement on most
extent based filesystems.

However, My conditions for merging such functionality into fallocate
are:
1. it needs xfs_io support [to provide]
2. comprehensive xfstests coverage, similar to the current
   hole punch coverage we have.

> >> #define EXT4_IOC_TRUNCATE_BLOCK_RANGE  _IOW('f', 18, struct truncate_range)
> >> struct truncate_range {
> >>  __u32 start_block;
> >>  __u32 length;
> >> };

And have 64 bit file size support, please!

Also FALLOC_FL_COLLAPSE_RANGE is probably a better name for the
operation being done ;)

Cheers,

Dave.
-- 
Dave Chinner
da...@fromorbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-23 Thread Eric Sandeen
On Jun 23, 2013, at 12:01 PM, Andreas Dilger  wrote:

> On 2013-06-23, at 0:07, Namjae Jeon  wrote:
> 
>> From: Namjae Jeon 
>> 
>> This patch series introduces 2 new ioctls for ext4.
>> 
>> Truncate_block_range ioctl truncates blocks from source file.
> 
> How is this different from fallocate(FALLOC_FL_PUNCH_HOLE)?  That is already 
> in existing kernels, and portable across multiple filesystems. 
> 
Same question.  Punch hole should do this already...


>> Transfer_block_range ioctl transfers data blocks from source file
>> and append them at the end of destination file.
> 
> There is already a similar ioctl for defragmenting files. Is it possible to 
> use that, or does it have different semantics?
> 
>> Ioctl1:EXT4_IOC_TRUNCATE_BLOCK_RANGE:
>> This ioctl truncates a range of data blocks from file.
>> It is useful to remove easily and quickly the garbage data
>> at the middle of file.
>> 
>> e.g. we have a movie file and there is long advertisement in movie file.
>> user want to remove only advertisement range.
> 
> While this works in theory, there is very little chance that the movie data 
> will align exactly to filesystem block boundaries. 
> 
Or more importantly on compression codec boundaries.   Wouldn't this look like 
corruption at playback time?

Eric

> Cheers, Andreas 
> 
>> 1) Movie file (8GB), There is the adverisement of 500MB size.
>> 
>> | |   | |
>> |   a) range  | b) Advertisement  | c) range| 
>> | | (unneeded data)   | |
>> |_|___|_| 
>> 
>> 2) Currently if user want to remove portion b), the conventional way
>>   would be to copy a) and c) (7.5GB) to new file by reading data from
>>   original file and writing to new file, followed up by  delete original
>>   file and rename new file. It will take long time.
>>   When we measure time, it takes around 3 minutes.
>> 
>> 3) If we use EXT4_IOC_TRUNCATE_BLOCK_RANGE, we can have garbage data removed
>>   in less than a second. Also, no need to perform deletion and rename.
>> ___
>> | ||
>> |   a) range  | c) range   | 
>> | ||
>> |_|| 
>> 
>> 
>> #define EXT4_IOC_TRUNCATE_BLOCK_RANGE  _IOW('f', 18, struct truncate_range)
>> struct truncate_range {
>>  __u32 start_block;
>>  __u32 length;
>> };
>> 
>> example =>
>> Originally the file "abc" has the below extent tree:
>> debugfs:  ex abc
>> Level Entries   LogicalPhysical Length Flags
>> 0/ 0   1/  3 0 - 4  33615 -  33619  5 
>> 0/ 0   2/  3 5 - 9  33855 -  33859  5 
>> 0/ 0   3/  310 -14  69657 -  69661  5
>> 
>> ls -lh abc
>> -rw-r--r--1 root 0  60.0K Jan  1 00:01 abc
>> 
>> du -h abc
>> 60.0Kabc
>> 
>> e4_truncate_block_range abc 2 10
>> Return:
>> : Success
>> 
>> After executing truncate_block_range ioctl, the extent tree:
>> ex abc
>> Level Entries   LogicalPhysical Length Flags
>> 0/ 0   1/  2 0 - 1  33615 -  33616  2 
>> 0/ 0   2/  2 2 - 4  69659 -  69661  3 
>> 
>> ls -lh abc
>> -rw-r--r--1 root 0  20.0K Jan  1 00:08 abc
>> 
>> du -h abc
>> 20.0Kabc
>> 
>> This ioctl works in 2 parts:
>> 1) remove _only_ data blocks that resides within specified range.
>> If the entire range is a hole than nothing is removed.
>> 
>> 2) update file's logical block offsets ranging from block number
>> "start_block + length" to last logical block of file such that
>> lblk_number = lblk_number - length;
>> This is done by updating starting block of all the extents that
>> resides within the range.
>> 
>> If "start_block + length" is already equal to the last block of file
>> than no block is updated. This case is similar to convential truncate.
>> 
>> In the above example:
>> The data blocks ranging from [2 - 11] have been removed
>> and the logical offsets of the file beyond block number 12 till last block 
>> of file are updated by subtracting length from each of logical numbers.
>> This gives a contiguous logical space to the file.
>> Also, the logical size and disksize of the file are updated accordingly.
>> 
>> Ioctl2:EXT4_IOC_TRANSFER_BLOCK_RANGE:
>> This ioctl transfers a range of data blocks from source file and append
>> them at the end of the destination file.
>> This is not actual data transfer but only metadata is moved.
>> 
>> 
>> | |   | |
>> |   a) range  | b) range  | c) range|
>> | |   |  

Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-23 Thread Andreas Dilger
On 2013-06-23, at 0:07, Namjae Jeon  wrote:

> From: Namjae Jeon 
> 
> This patch series introduces 2 new ioctls for ext4.
> 
> Truncate_block_range ioctl truncates blocks from source file.

How is this different from fallocate(FALLOC_FL_PUNCH_HOLE)?  That is already in 
existing kernels, and portable across multiple filesystems. 

> Transfer_block_range ioctl transfers data blocks from source file
> and append them at the end of destination file.

There is already a similar ioctl for defragmenting files. Is it possible to use 
that, or does it have different semantics?

> Ioctl1:EXT4_IOC_TRUNCATE_BLOCK_RANGE:
> This ioctl truncates a range of data blocks from file.
> It is useful to remove easily and quickly the garbage data
> at the middle of file.
> 
> e.g. we have a movie file and there is long advertisement in movie file.
> user want to remove only advertisement range.

While this works in theory, there is very little chance that the movie data 
will align exactly to filesystem block boundaries. 

Cheers, Andreas 

> 1) Movie file (8GB), There is the adverisement of 500MB size.
> 
> | |   | |
> |   a) range  | b) Advertisement  | c) range| 
> | | (unneeded data)   | |
> |_|___|_| 
> 
> 2) Currently if user want to remove portion b), the conventional way
>would be to copy a) and c) (7.5GB) to new file by reading data from
>original file and writing to new file, followed up by  delete original
>file and rename new file. It will take long time.
>When we measure time, it takes around 3 minutes.
> 
> 3) If we use EXT4_IOC_TRUNCATE_BLOCK_RANGE, we can have garbage data removed
>in less than a second. Also, no need to perform deletion and rename.
> ___
> | ||
> |   a) range  | c) range   | 
> | ||
> |_|| 
> 
> 
> #define EXT4_IOC_TRUNCATE_BLOCK_RANGE  _IOW('f', 18, struct truncate_range)
> struct truncate_range {
>   __u32 start_block;
>   __u32 length;
> };
> 
> example =>
> Originally the file "abc" has the below extent tree:
> debugfs:  ex abc
> Level Entries   LogicalPhysical Length Flags
> 0/ 0   1/  3 0 - 4  33615 -  33619  5 
> 0/ 0   2/  3 5 - 9  33855 -  33859  5 
> 0/ 0   3/  310 -14  69657 -  69661  5
> 
> ls -lh abc
> -rw-r--r--1 root 0  60.0K Jan  1 00:01 abc
> 
> du -h abc
> 60.0Kabc
> 
> e4_truncate_block_range abc 2 10
> Return:
> : Success
> 
> After executing truncate_block_range ioctl, the extent tree:
> ex abc
> Level Entries   LogicalPhysical Length Flags
> 0/ 0   1/  2 0 - 1  33615 -  33616  2 
> 0/ 0   2/  2 2 - 4  69659 -  69661  3 
> 
> ls -lh abc
> -rw-r--r--1 root 0  20.0K Jan  1 00:08 abc
> 
> du -h abc
> 20.0Kabc
> 
> This ioctl works in 2 parts:
> 1) remove _only_ data blocks that resides within specified range.
> If the entire range is a hole than nothing is removed.
> 
> 2) update file's logical block offsets ranging from block number
> "start_block + length" to last logical block of file such that
> lblk_number = lblk_number - length;
> This is done by updating starting block of all the extents that
> resides within the range.
> 
> If "start_block + length" is already equal to the last block of file
> than no block is updated. This case is similar to convential truncate.
> 
> In the above example:
> The data blocks ranging from [2 - 11] have been removed
> and the logical offsets of the file beyond block number 12 till last block 
> of file are updated by subtracting length from each of logical numbers.
> This gives a contiguous logical space to the file.
> Also, the logical size and disksize of the file are updated accordingly.
> 
> Ioctl2:EXT4_IOC_TRANSFER_BLOCK_RANGE:
> This ioctl transfers a range of data blocks from source file and append
> them at the end of the destination file.
> This is not actual data transfer but only metadata is moved.
> 
> 
> | |   | |
> |   a) range  | b) range  | c) range|
> | |   | |
> |_|___|_|
> 
> If user does not want b) in the orig file but wants to make a new file
> comprising only b) OR wants b) at the end of an already existing file,
> the conventional way of doing it would be to:
> 1) Copy b) to new file
> 2) Copy c) to temp file
> 3) Truncate orig 

Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-23 Thread Namjae Jeon
Hi.

I attached testcases for these ioctls.

Thanks.

2013/6/23 Namjae Jeon :
> From: Namjae Jeon 
>
> This patch series introduces 2 new ioctls for ext4.
>
> Truncate_block_range ioctl truncates blocks from source file.
> Transfer_block_range ioctl transfers data blocks from source file
> and append them at the end of destination file.
>
> Ioctl1: EXT4_IOC_TRUNCATE_BLOCK_RANGE:
> This ioctl truncates a range of data blocks from file.
> It is useful to remove easily and quickly the garbage data
> at the middle of file.
>
> e.g. we have a movie file and there is long advertisement in movie file.
> user want to remove only advertisement range.
>
>  1) Movie file (8GB), There is the adverisement of 500MB size.
>  
>  | |   | |
>  |   a) range  | b) Advertisement  | c) range|
>  | | (unneeded data)   | |
>  |_|___|_|
>
>  2) Currently if user want to remove portion b), the conventional way
> would be to copy a) and c) (7.5GB) to new file by reading data from
> original file and writing to new file, followed up by  delete original
> file and rename new file. It will take long time.
> When we measure time, it takes around 3 minutes.
>
>  3) If we use EXT4_IOC_TRUNCATE_BLOCK_RANGE, we can have garbage data removed
> in less than a second. Also, no need to perform deletion and rename.
>  ___
>  | ||
>  |   a) range  | c) range   |
>  | ||
>  |_||
>
>
> #define EXT4_IOC_TRUNCATE_BLOCK_RANGE  _IOW('f', 18, struct truncate_range)
> struct truncate_range {
>__u32 start_block;
>__u32 length;
> };
>
> example =>
> Originally the file "abc" has the below extent tree:
> debugfs:  ex abc
> Level Entries   LogicalPhysical Length Flags
>  0/ 0   1/  3 0 - 4  33615 -  33619  5
>  0/ 0   2/  3 5 - 9  33855 -  33859  5
>  0/ 0   3/  310 -14  69657 -  69661  5
>
> ls -lh abc
> -rw-r--r--1 root 0  60.0K Jan  1 00:01 abc
>
> du -h abc
> 60.0K   abc
>
> e4_truncate_block_range abc 2 10
> Return:
> : Success
>
> After executing truncate_block_range ioctl, the extent tree:
> ex abc
> Level Entries   LogicalPhysical Length Flags
>  0/ 0   1/  2 0 - 1  33615 -  33616  2
>  0/ 0   2/  2 2 - 4  69659 -  69661  3
>
> ls -lh abc
> -rw-r--r--1 root 0  20.0K Jan  1 00:08 abc
>
> du -h abc
> 20.0K   abc
>
> This ioctl works in 2 parts:
> 1) remove _only_ data blocks that resides within specified range.
> If the entire range is a hole than nothing is removed.
>
> 2) update file's logical block offsets ranging from block number
> "start_block + length" to last logical block of file such that
> lblk_number = lblk_number - length;
> This is done by updating starting block of all the extents that
> resides within the range.
>
> If "start_block + length" is already equal to the last block of file
> than no block is updated. This case is similar to convential truncate.
>
> In the above example:
> The data blocks ranging from [2 - 11] have been removed
> and the logical offsets of the file beyond block number 12 till last block
> of file are updated by subtracting length from each of logical numbers.
> This gives a contiguous logical space to the file.
> Also, the logical size and disksize of the file are updated accordingly.
>
> Ioctl2: EXT4_IOC_TRANSFER_BLOCK_RANGE:
> This ioctl transfers a range of data blocks from source file and append
> them at the end of the destination file.
> This is not actual data transfer but only metadata is moved.
>
>  
>  | |   | |
>  |   a) range  | b) range  | c) range|
>  | |   | |
>  |_|___|_|
>
> If user does not want b) in the orig file but wants to make a new file
> comprising only b) OR wants b) at the end of an already existing file,
> the conventional way of doing it would be to:
> 1) Copy b) to new file
> 2) Copy c) to temp file
> 3) Truncate orig file to a)
> 4) Copy c) from temp file to the end of orig file.
> 5) Delete temp file.
>
> After this operations =>
> orig_file:
> __
> | |   |
> |   a) range  | c) range  |
> | |   |
> |_|___|
>
> new_file:
> 

Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-23 Thread Namjae Jeon
Hi.

I attached testcases for these ioctls.

Thanks.

2013/6/23 Namjae Jeon linkinj...@gmail.com:
 From: Namjae Jeon namjae.j...@samsung.com

 This patch series introduces 2 new ioctls for ext4.

 Truncate_block_range ioctl truncates blocks from source file.
 Transfer_block_range ioctl transfers data blocks from source file
 and append them at the end of destination file.

 Ioctl1: EXT4_IOC_TRUNCATE_BLOCK_RANGE:
 This ioctl truncates a range of data blocks from file.
 It is useful to remove easily and quickly the garbage data
 at the middle of file.

 e.g. we have a movie file and there is long advertisement in movie file.
 user want to remove only advertisement range.

  1) Movie file (8GB), There is the adverisement of 500MB size.
  
  | |   | |
  |   a) range  | b) Advertisement  | c) range|
  | | (unneeded data)   | |
  |_|___|_|

  2) Currently if user want to remove portion b), the conventional way
 would be to copy a) and c) (7.5GB) to new file by reading data from
 original file and writing to new file, followed up by  delete original
 file and rename new file. It will take long time.
 When we measure time, it takes around 3 minutes.

  3) If we use EXT4_IOC_TRUNCATE_BLOCK_RANGE, we can have garbage data removed
 in less than a second. Also, no need to perform deletion and rename.
  ___
  | ||
  |   a) range  | c) range   |
  | ||
  |_||


 #define EXT4_IOC_TRUNCATE_BLOCK_RANGE  _IOW('f', 18, struct truncate_range)
 struct truncate_range {
__u32 start_block;
__u32 length;
 };

 example =
 Originally the file abc has the below extent tree:
 debugfs:  ex abc
 Level Entries   LogicalPhysical Length Flags
  0/ 0   1/  3 0 - 4  33615 -  33619  5
  0/ 0   2/  3 5 - 9  33855 -  33859  5
  0/ 0   3/  310 -14  69657 -  69661  5

 ls -lh abc
 -rw-r--r--1 root 0  60.0K Jan  1 00:01 abc

 du -h abc
 60.0K   abc

 e4_truncate_block_range abc 2 10
 Return:
 : Success

 After executing truncate_block_range ioctl, the extent tree:
 ex abc
 Level Entries   LogicalPhysical Length Flags
  0/ 0   1/  2 0 - 1  33615 -  33616  2
  0/ 0   2/  2 2 - 4  69659 -  69661  3

 ls -lh abc
 -rw-r--r--1 root 0  20.0K Jan  1 00:08 abc

 du -h abc
 20.0K   abc

 This ioctl works in 2 parts:
 1) remove _only_ data blocks that resides within specified range.
 If the entire range is a hole than nothing is removed.

 2) update file's logical block offsets ranging from block number
 start_block + length to last logical block of file such that
 lblk_number = lblk_number - length;
 This is done by updating starting block of all the extents that
 resides within the range.

 If start_block + length is already equal to the last block of file
 than no block is updated. This case is similar to convential truncate.

 In the above example:
 The data blocks ranging from [2 - 11] have been removed
 and the logical offsets of the file beyond block number 12 till last block
 of file are updated by subtracting length from each of logical numbers.
 This gives a contiguous logical space to the file.
 Also, the logical size and disksize of the file are updated accordingly.

 Ioctl2: EXT4_IOC_TRANSFER_BLOCK_RANGE:
 This ioctl transfers a range of data blocks from source file and append
 them at the end of the destination file.
 This is not actual data transfer but only metadata is moved.

  
  | |   | |
  |   a) range  | b) range  | c) range|
  | |   | |
  |_|___|_|

 If user does not want b) in the orig file but wants to make a new file
 comprising only b) OR wants b) at the end of an already existing file,
 the conventional way of doing it would be to:
 1) Copy b) to new file
 2) Copy c) to temp file
 3) Truncate orig file to a)
 4) Copy c) from temp file to the end of orig file.
 5) Delete temp file.

 After this operations =
 orig_file:
 __
 | |   |
 |   a) range  | c) range  |
 | |   |
 |_|___|

 new_file:
 ___
 | |
 |   b) range  |
 | |
 

Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-23 Thread Andreas Dilger
On 2013-06-23, at 0:07, Namjae Jeon linkinj...@gmail.com wrote:

 From: Namjae Jeon namjae.j...@samsung.com
 
 This patch series introduces 2 new ioctls for ext4.
 
 Truncate_block_range ioctl truncates blocks from source file.

How is this different from fallocate(FALLOC_FL_PUNCH_HOLE)?  That is already in 
existing kernels, and portable across multiple filesystems. 

 Transfer_block_range ioctl transfers data blocks from source file
 and append them at the end of destination file.

There is already a similar ioctl for defragmenting files. Is it possible to use 
that, or does it have different semantics?

 Ioctl1:EXT4_IOC_TRUNCATE_BLOCK_RANGE:
 This ioctl truncates a range of data blocks from file.
 It is useful to remove easily and quickly the garbage data
 at the middle of file.
 
 e.g. we have a movie file and there is long advertisement in movie file.
 user want to remove only advertisement range.

While this works in theory, there is very little chance that the movie data 
will align exactly to filesystem block boundaries. 

Cheers, Andreas 

 1) Movie file (8GB), There is the adverisement of 500MB size.
 
 | |   | |
 |   a) range  | b) Advertisement  | c) range| 
 | | (unneeded data)   | |
 |_|___|_| 
 
 2) Currently if user want to remove portion b), the conventional way
would be to copy a) and c) (7.5GB) to new file by reading data from
original file and writing to new file, followed up by  delete original
file and rename new file. It will take long time.
When we measure time, it takes around 3 minutes.
 
 3) If we use EXT4_IOC_TRUNCATE_BLOCK_RANGE, we can have garbage data removed
in less than a second. Also, no need to perform deletion and rename.
 ___
 | ||
 |   a) range  | c) range   | 
 | ||
 |_|| 
 
 
 #define EXT4_IOC_TRUNCATE_BLOCK_RANGE  _IOW('f', 18, struct truncate_range)
 struct truncate_range {
   __u32 start_block;
   __u32 length;
 };
 
 example =
 Originally the file abc has the below extent tree:
 debugfs:  ex abc
 Level Entries   LogicalPhysical Length Flags
 0/ 0   1/  3 0 - 4  33615 -  33619  5 
 0/ 0   2/  3 5 - 9  33855 -  33859  5 
 0/ 0   3/  310 -14  69657 -  69661  5
 
 ls -lh abc
 -rw-r--r--1 root 0  60.0K Jan  1 00:01 abc
 
 du -h abc
 60.0Kabc
 
 e4_truncate_block_range abc 2 10
 Return:
 : Success
 
 After executing truncate_block_range ioctl, the extent tree:
 ex abc
 Level Entries   LogicalPhysical Length Flags
 0/ 0   1/  2 0 - 1  33615 -  33616  2 
 0/ 0   2/  2 2 - 4  69659 -  69661  3 
 
 ls -lh abc
 -rw-r--r--1 root 0  20.0K Jan  1 00:08 abc
 
 du -h abc
 20.0Kabc
 
 This ioctl works in 2 parts:
 1) remove _only_ data blocks that resides within specified range.
 If the entire range is a hole than nothing is removed.
 
 2) update file's logical block offsets ranging from block number
 start_block + length to last logical block of file such that
 lblk_number = lblk_number - length;
 This is done by updating starting block of all the extents that
 resides within the range.
 
 If start_block + length is already equal to the last block of file
 than no block is updated. This case is similar to convential truncate.
 
 In the above example:
 The data blocks ranging from [2 - 11] have been removed
 and the logical offsets of the file beyond block number 12 till last block 
 of file are updated by subtracting length from each of logical numbers.
 This gives a contiguous logical space to the file.
 Also, the logical size and disksize of the file are updated accordingly.
 
 Ioctl2:EXT4_IOC_TRANSFER_BLOCK_RANGE:
 This ioctl transfers a range of data blocks from source file and append
 them at the end of the destination file.
 This is not actual data transfer but only metadata is moved.
 
 
 | |   | |
 |   a) range  | b) range  | c) range|
 | |   | |
 |_|___|_|
 
 If user does not want b) in the orig file but wants to make a new file
 comprising only b) OR wants b) at the end of an already existing file,
 the conventional way of doing it would be to:
 1) Copy b) to new file
 2) Copy c) to temp file
 3) Truncate orig file to a)
 4) Copy c) from temp file to the end of orig file.
 5) Delete 

Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-23 Thread Eric Sandeen
On Jun 23, 2013, at 12:01 PM, Andreas Dilger adil...@dilger.ca wrote:

 On 2013-06-23, at 0:07, Namjae Jeon linkinj...@gmail.com wrote:
 
 From: Namjae Jeon namjae.j...@samsung.com
 
 This patch series introduces 2 new ioctls for ext4.
 
 Truncate_block_range ioctl truncates blocks from source file.
 
 How is this different from fallocate(FALLOC_FL_PUNCH_HOLE)?  That is already 
 in existing kernels, and portable across multiple filesystems. 
 
Same question.  Punch hole should do this already...


 Transfer_block_range ioctl transfers data blocks from source file
 and append them at the end of destination file.
 
 There is already a similar ioctl for defragmenting files. Is it possible to 
 use that, or does it have different semantics?
 
 Ioctl1:EXT4_IOC_TRUNCATE_BLOCK_RANGE:
 This ioctl truncates a range of data blocks from file.
 It is useful to remove easily and quickly the garbage data
 at the middle of file.
 
 e.g. we have a movie file and there is long advertisement in movie file.
 user want to remove only advertisement range.
 
 While this works in theory, there is very little chance that the movie data 
 will align exactly to filesystem block boundaries. 
 
Or more importantly on compression codec boundaries.   Wouldn't this look like 
corruption at playback time?

Eric

 Cheers, Andreas 
 
 1) Movie file (8GB), There is the adverisement of 500MB size.
 
 | |   | |
 |   a) range  | b) Advertisement  | c) range| 
 | | (unneeded data)   | |
 |_|___|_| 
 
 2) Currently if user want to remove portion b), the conventional way
   would be to copy a) and c) (7.5GB) to new file by reading data from
   original file and writing to new file, followed up by  delete original
   file and rename new file. It will take long time.
   When we measure time, it takes around 3 minutes.
 
 3) If we use EXT4_IOC_TRUNCATE_BLOCK_RANGE, we can have garbage data removed
   in less than a second. Also, no need to perform deletion and rename.
 ___
 | ||
 |   a) range  | c) range   | 
 | ||
 |_|| 
 
 
 #define EXT4_IOC_TRUNCATE_BLOCK_RANGE  _IOW('f', 18, struct truncate_range)
 struct truncate_range {
  __u32 start_block;
  __u32 length;
 };
 
 example =
 Originally the file abc has the below extent tree:
 debugfs:  ex abc
 Level Entries   LogicalPhysical Length Flags
 0/ 0   1/  3 0 - 4  33615 -  33619  5 
 0/ 0   2/  3 5 - 9  33855 -  33859  5 
 0/ 0   3/  310 -14  69657 -  69661  5
 
 ls -lh abc
 -rw-r--r--1 root 0  60.0K Jan  1 00:01 abc
 
 du -h abc
 60.0Kabc
 
 e4_truncate_block_range abc 2 10
 Return:
 : Success
 
 After executing truncate_block_range ioctl, the extent tree:
 ex abc
 Level Entries   LogicalPhysical Length Flags
 0/ 0   1/  2 0 - 1  33615 -  33616  2 
 0/ 0   2/  2 2 - 4  69659 -  69661  3 
 
 ls -lh abc
 -rw-r--r--1 root 0  20.0K Jan  1 00:08 abc
 
 du -h abc
 20.0Kabc
 
 This ioctl works in 2 parts:
 1) remove _only_ data blocks that resides within specified range.
 If the entire range is a hole than nothing is removed.
 
 2) update file's logical block offsets ranging from block number
 start_block + length to last logical block of file such that
 lblk_number = lblk_number - length;
 This is done by updating starting block of all the extents that
 resides within the range.
 
 If start_block + length is already equal to the last block of file
 than no block is updated. This case is similar to convential truncate.
 
 In the above example:
 The data blocks ranging from [2 - 11] have been removed
 and the logical offsets of the file beyond block number 12 till last block 
 of file are updated by subtracting length from each of logical numbers.
 This gives a contiguous logical space to the file.
 Also, the logical size and disksize of the file are updated accordingly.
 
 Ioctl2:EXT4_IOC_TRANSFER_BLOCK_RANGE:
 This ioctl transfers a range of data blocks from source file and append
 them at the end of the destination file.
 This is not actual data transfer but only metadata is moved.
 
 
 | |   | |
 |   a) range  | b) range  | c) range|
 | |   | |
 |_|___|_|
 
 If user does not want b) in the orig file but wants to make a new file
 

Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-23 Thread Dave Chinner
On Sun, Jun 23, 2013 at 08:32:32PM -0400, Eric Sandeen wrote:
 On Jun 23, 2013, at 12:01 PM, Andreas Dilger adil...@dilger.ca wrote:
 
  On 2013-06-23, at 0:07, Namjae Jeon linkinj...@gmail.com wrote:
  
  From: Namjae Jeon namjae.j...@samsung.com
  
  This patch series introduces 2 new ioctls for ext4.
  
  Truncate_block_range ioctl truncates blocks from source file.
  
  How is this different from fallocate(FALLOC_FL_PUNCH_HOLE)?  That is 
  already in existing kernels, and portable across multiple filesystems. 
  
 Same question.  Punch hole should do this already...

Hole punch doesn't change the offsets of subsequent extents in the
file - it leaves a hole. This doesn't leave a hole at all - all the 
extents above the range are shifted down to offset where the extents
being punched out started.

  Transfer_block_range ioctl transfers data blocks from source file
  and append them at the end of destination file.
  
  There is already a similar ioctl for defragmenting files. Is it possible to 
  use that, or does it have different semantics?
  
  Ioctl1:EXT4_IOC_TRUNCATE_BLOCK_RANGE:
  This ioctl truncates a range of data blocks from file.
  It is useful to remove easily and quickly the garbage data
  at the middle of file.
  
  e.g. we have a movie file and there is long advertisement in movie file.
  user want to remove only advertisement range.
  
  While this works in theory, there is very little chance that the movie data 
  will align exactly to filesystem block boundaries. 
  
 Or more importantly on compression codec boundaries.   Wouldn't this look 
 like corruption at playback time?

Not necessarily. Video codecs are encapsulated in a container that
can be used to link key frames together so you can do this sort of
manipulation of the file contents and just change an
offset-to-next-keyframe value in the container and it all just
works. Non linear editting (NLE) software has been doing this
manually for 15 years by copying data around - this just optimises
the operation by manipulating the extent mapping rather than needing
to physically copy the data.

FWIW, I've heard persistent rumors going back several years of
various DVR companies shipping equivalent ioctl-based functionality
for XFS filesystems to both insert and remove chunks in video
streams, but I've never been able to find the code for it anywhere.

Hence, at minimum, this should be a fallocate() operation, not a ext4
specific ioctl as it is relatively trivial to implement on most
extent based filesystems.

However, My conditions for merging such functionality into fallocate
are:
1. it needs xfs_io support [to provide]
2. comprehensive xfstests coverage, similar to the current
   hole punch coverage we have.

  #define EXT4_IOC_TRUNCATE_BLOCK_RANGE  _IOW('f', 18, struct truncate_range)
  struct truncate_range {
   __u32 start_block;
   __u32 length;
  };

And have 64 bit file size support, please!

Also FALLOC_FL_COLLAPSE_RANGE is probably a better name for the
operation being done ;)

Cheers,

Dave.
-- 
Dave Chinner
da...@fromorbit.com
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH 0/3] ext4: introduce two new ioctls

2013-06-23 Thread Theodore Ts'o
On Mon, Jun 24, 2013 at 12:44:59PM +1000, Dave Chinner wrote:
 
 Hence, at minimum, this should be a fallocate() operation, not a ext4
 specific ioctl as it is relatively trivial to implement on most
 extent based filesystems.

The fallocate() uses a units of bytes for the offset and length; would
a FALLOC_FL_COLLAPSE_RANGE be guaranteed to work on any arbitrary
offset and length?  Or would it only work if the offset and length are
multiples of the file system blocksize?

The the EXT4_IOC_TRUNCATE_BLOCK_RANGE interface solves this problem by
using units of file system blocks (i.e., __u32 start_block), but that
raises another issue, which is it forces the user space program to
somehow figure out the file system block size, which seems a bit nasty.

   - Ted
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/