On Tue, Oct 1, 2013 at 1:31 PM, Roland Mainz <[email protected]> wrote: > On Tue, Oct 1, 2013 at 5:50 AM, Roland Mainz <[email protected]> wrote: >> On Mon, Sep 30, 2013 at 7:06 PM, Roland Mainz <[email protected]> >> wrote: >>> On Mon, Sep 30, 2013 at 5:12 PM, Roland Mainz <[email protected]> >>> wrote: >>>> On Mon, Sep 30, 2013 at 4:17 PM, Dan Shelton <[email protected]> >>>> wrote: >>>>> I'm just forwarding the old conversation as a reminder - AST pax still >>>>> does not support SEEK_HOLE or SEEK_DATA (nor does it SUN.holesdata pax >>>>> header), nor do AST cp and mv support files with holes. >>>>> >>>>> As consequence neither AST pax, cp or mv are competitive to any of >>>>> such implementations which support SEEK_HOLE and SEEK_DATA. >>>>> >>>>> For example moving a 200GB file with 99% holes with GNU mv (GNU >>>>> supports SEEK_HOLE/SEEK_DATA since 2010) across filesystem takes less >>>>> than a 4 seconds with GNU mv but takes a WHOPPING 18 minutes with AST >>>>> mv. >>>> [snip] >>>> >>>> Erm... >>>> ... AFAIK a copy-file-data algorithm would be just this: >>>> 1. Test whether $ getconf MIN_HOLE_SIZE <srcpath> # returns a value > 0 >>>> 2. If [1] is true then check whether the file has at least one hole >>>> (via |SEEK_HOLE|) >>>> 3. If [2] is true switch to a special version of the data copying code >>>> which "simply" copies data via |write()| until it hits a hole and then >>>> uses |lseek()| to seek forward to the next position and then uses >>>> |write()| again. >>>> >>>> Glenn: Does that sound correct ? >>> >>> Some notes for myself: >> [snip] >> >> Attached (as "astksh20130926_sparsefile_cp001.diff.txt") is a >> prototype patch which adds sparse file support to AST >> cp(1)/mv(1)/ln(1) via using the |SEEK_HOLE|/|SEEK_DATA| API from >> POSIX. >> Additionally I've attached "lsholes.c.txt" which is a small test >> application to show the hole/data layout of a sparse file. >> >> * Notes: >> - |sfmove()| seems to turn longer sequences of '\0\ data into holes. >> While this is usefull _sometimes_ its devastating for databases&&other >> software which depend on an exact replication of the layout of the >> holes (and real data which are mostly of the value '\0'. >> - Erm... it's 5:35h AM here... any idea how |sfmove()| figures out if >> data are all zero bytes and should be skipped ? > [snip] > > It turns out there is a sfio flag called |SF_WHOLE| to disable the > "turn zero bytes into hols"-functionality... > ... attached (as "astksh20130926_sparsefile_cp002.diff.txt") is a > patch which exactly does that... > > ... AFAIK only open question before commiting it to ast-ksh is the > --sparse option and how to name the keys for it...
We've tested your patch with ORACLE databases and your own software which creates sparse files and I'd say I'm impressed. It works better than expected and sheds a bad light on other cp(1) versions - it looks so "simple" but yet nobody did implement it for cp(1). What I dislike is to have an option to "invent" holes out of 0 byte sequences. That is a quite dangerous way of blowing up things badly and should be done in a separate utility. Irek _______________________________________________ ast-developers mailing list [email protected] http://lists.research.att.com/mailman/listinfo/ast-developers
