Re: svn_io_file_flush_to_disk fails on AFS filesystems on Darwin

[Branko Čibej]

On 26.06.2019 18:41, Nathan Hartman wrote:
> On Wed, Jun 26, 2019 at 11:49 AM Branko Čibej  > wrote:
>
> On 26.06.2019 04:52, Quentin Smith wrote:
> > svn_io_file_flush_to_disk in subversion/libsvn_subr/io.c fails to
> > flush files on AFS filesystems on Darwin. The user-visible
> experience is:
>
>
> (snip)
>
> > The code ignores EINVAL with a comment about filesystems that don't
> > support it; evidently on Darwin this also needs to include ENOTTY.
> >
> > (Also, I suspect it should fall back from fcntl to fsync, since I
> > suspect fsync /does/ work on AFS filesystems.)
>
>
> (snip)
>
> Last but not least, I do wish filesystems were consistent in their
> implementation ... what on earth is ENOTTY doing here?
>
> -- Brane
>
>
> Looks like libuv encountered similar issues and applied similar
> handling to that in SQLite:
>
> https://github.com/libuv/libuv/commit/5b0e1d75a207bb1c662f4495c0d8ba1e81a5bb7d
>

Something tell me this should be solved in APR ...

Re: Unicode composable characters on macOS [was: Subversion 2.0]

[Thomas Singer]

Hi Branko,

Thanks for the detailed explanation. Would you mind to add the 
description to the linked issue and mark it as 
resolved/works-for-me/no-bug, so this information is not lost?


--
Best regards,
Thomas Singer
=
syntevo GmbH


On 26/06/2019 17:39, Branko Čibej wrote:

On 26.06.2019 10:40, Marc Strapetz wrote:

On 25.06.2019 23:35, Branko Čibej wrote:> On 25.06.2019 19:15, Thomas
Singer wrote:

What I don't like:
- after more than a decade the umlaut problem of composed/decomposed
UTF-8 has not been solved


It has, actually, in Apple's APFS, where the fix belongs.


That sounds interesting. Just to be sure, you are referring to this
problem:

https://issues.apache.org/jira/browse/SVN-2464

? It would be great to have some more information for which OSX
version and which file systems the problem should be resolved.



The original problem was that Apples HFS+ filesystem normalized paths to
Unicode Normalisation Form D. In practice that meant that if you created
a file with a name that contained a composable character, then read that
name from the filesystem, you could get different results (i.e., the
name was "the same" as far as Unicode normalisation is concerned, but
the actual representation bytes were different).

The new APFS filesystem (which is the default in the last two versions
of macOS, IIRC) doesn't do that any more.

This is on local disk, which is APFS:

brane@zulu:~/src/svn/test$ svnadmin create repo
brane@zulu:~/src/svn/test$ svn co file://$(pwd)/repo wc
Checked out revision 0.
brane@zulu:~/src/svn/test$ touch wc/čibej
brane@zulu:~/src/svn/test$ svn add wc/čibej
A wc/čibej
brane@zulu:~/src/svn/test$ svn st wc/
A   wc/čibej

and this is on an HFS+ disk image:

brane@zulu:/Volumes/hfs$ svnadmin create repo
brane@zulu:/Volumes/hfs$ svn co file://$(pwd)/repo wc
Checked out revision 0.
brane@zulu:/Volumes/hfs$ touch wc/čibej
brane@zulu:/Volumes/hfs$ svn add wc/čibej
A wc/čibej
brane@zulu:/Volumes/hfs$ svn st wc/
?   wc/čibej
!   wc/čibej

The second instance clearly shows that the filesystem changed the file name.

-- Brane

Re: svn_io_file_flush_to_disk fails on AFS filesystems on Darwin

[Nathan Hartman]

On Wed, Jun 26, 2019 at 11:49 AM Branko Čibej  wrote:

> On 26.06.2019 04:52, Quentin Smith wrote:
> > svn_io_file_flush_to_disk in subversion/libsvn_subr/io.c fails to
> > flush files on AFS filesystems on Darwin. The user-visible experience is:


(snip)

> The code ignores EINVAL with a comment about filesystems that don't
> > support it; evidently on Darwin this also needs to include ENOTTY.
> >
> > (Also, I suspect it should fall back from fcntl to fsync, since I
> > suspect fsync /does/ work on AFS filesystems.)


(snip)

Last but not least, I do wish filesystems were consistent in their
> implementation ... what on earth is ENOTTY doing here?
>
> -- Brane


Looks like libuv encountered similar issues and applied similar handling to
that in SQLite:

https://github.com/libuv/libuv/commit/5b0e1d75a207bb1c662f4495c0d8ba1e81a5bb7d

Re: svn_io_file_flush_to_disk fails on AFS filesystems on Darwin

[Branko Čibej]

On 26.06.2019 04:52, Quentin Smith wrote:
> Hi,
>
> svn_io_file_flush_to_disk in subversion/libsvn_subr/io.c fails to
> flush files on AFS filesystems on Darwin. The user-visible experience is:
>
> $ svn commit
> svn: E25: Commit failed (details follow):
> svn: E25: Can't write '/afs/path/db/txn-current' atomically
> svn: E25: Can't flush file '/afs/path/db/svn-CNFY6N' to disk:
> Inappropriate ioctl for device
>
> This is because Darwin defines F_FULLFSYNC, which on AFS (and maybe
> other filesystems?) returns ENOTTY.
>
> The code ignores EINVAL with a comment about filesystems that don't
> support it; evidently on Darwin this also needs to include ENOTTY.
>
> (Also, I suspect it should fall back from fcntl to fsync, since I
> suspect fsync /does/ work on AFS filesystems.)
>
> I repro'd this with svn 1.10.3 (r1842928), but inspection of trunk
> confirms the code has not changed.

First of all, thanks for the repor.

Please provide a standalone reproduction script, using our command-line
client.

Last but not least, I do wish filesystems were consistent in their
implementation ... what on earth is ENOTTY doing here?

-- Brane

Unicode composable characters on macOS [was: Subversion 2.0]

[Branko Čibej]

On 26.06.2019 10:40, Marc Strapetz wrote:
> On 25.06.2019 23:35, Branko Čibej wrote:> On 25.06.2019 19:15, Thomas
> Singer wrote:
>>> What I don't like:
>>> - after more than a decade the umlaut problem of composed/decomposed
>>> UTF-8 has not been solved
>>
>> It has, actually, in Apple's APFS, where the fix belongs.
>
> That sounds interesting. Just to be sure, you are referring to this
> problem:
>
> https://issues.apache.org/jira/browse/SVN-2464
>
> ? It would be great to have some more information for which OSX
> version and which file systems the problem should be resolved.


The original problem was that Apples HFS+ filesystem normalized paths to
Unicode Normalisation Form D. In practice that meant that if you created
a file with a name that contained a composable character, then read that
name from the filesystem, you could get different results (i.e., the
name was "the same" as far as Unicode normalisation is concerned, but
the actual representation bytes were different).

The new APFS filesystem (which is the default in the last two versions
of macOS, IIRC) doesn't do that any more.

This is on local disk, which is APFS:

brane@zulu:~/src/svn/test$ svnadmin create repo
brane@zulu:~/src/svn/test$ svn co file://$(pwd)/repo wc
Checked out revision 0.
brane@zulu:~/src/svn/test$ touch wc/čibej
brane@zulu:~/src/svn/test$ svn add wc/čibej 
A wc/čibej
brane@zulu:~/src/svn/test$ svn st wc/
A   wc/čibej 

and this is on an HFS+ disk image:

brane@zulu:/Volumes/hfs$ svnadmin create repo
brane@zulu:/Volumes/hfs$ svn co file://$(pwd)/repo wc
Checked out revision 0.
brane@zulu:/Volumes/hfs$ touch wc/čibej
brane@zulu:/Volumes/hfs$ svn add wc/čibej 
A wc/čibej
brane@zulu:/Volumes/hfs$ svn st wc/
?   wc/čibej
!   wc/čibej 

The second instance clearly shows that the filesystem changed the file name.

-- Brane

Re: Subversion 2.0

[Marc Strapetz]

On 25.06.2019 23:35, Branko Čibej wrote:> On 25.06.2019 19:15, Thomas 
Singer wrote:

What I don't like:
- after more than a decade the umlaut problem of composed/decomposed
UTF-8 has not been solved


It has, actually, in Apple's APFS, where the fix belongs.


That sounds interesting. Just to be sure, you are referring to this problem:

https://issues.apache.org/jira/browse/SVN-2464

? It would be great to have some more information for which OSX version 
and which file systems the problem should be resolved.


-Marc


On 25.06.2019 23:35, Branko Čibej wrote:

On 25.06.2019 19:15, Thomas Singer wrote:

What I don't like:
- after more than a decade the umlaut problem of composed/decomposed
UTF-8 has not been solved


It has, actually, in Apple's APFS, where the fix belongs.

-- Brane

Re: Subversion's community health

[Thomas Singer]

I agree to all you wrote. But the increased speed of releasing new 
versions with unstable experimental features somehow contradicts the 
otherwise mature development state. As a user of the JavaHL API we only 
consider a feature as stable if it is reflected in all binding APIs, too.


--
Best regards,
Thomas Singer
=
syntevo GmbH
https://www.syntevo.com
https://www.syntevo.com/blog


On 2019-06-14 16:34, Eric S. Raymond wrote:

Julian Foad :

Anyone with constructive suggestions, please do share them. Please let us
not dwell on our sadness and criticism of what went before; let us try to
keep this thread focused on positive solutions for what to do next.


You guys know me.  I'm a past contributor, occasional critic, often a
supporter. I did my best to push back when Linus Torvalds accused this
crew of incompetence.  And I, too, have had the recent experience of
watching a project I was hugely invested in - GPSD - slide into a
semi-active maintainence mode.

The main piece of advice I have for all of you is that you should
keep your expectations about Subversion's future realistic.

The brute fact is that git has taken over the version-control world.
It has stomped flat a couple of sttempts to compete with it in DVCS -
Mercurial, bzr, monotone. And Subversion is at a massive disdavantage
relative to *any* DVCS for reasons that should be too obvious to
need repeating.

Does Subversion have a future at all?  I think the answer is "Yes",
but it's not an exciting, sexy future.  You guys have only two selling
points I can see for new installations (1) Subversion's UI is
*massively* simpler than git's, and (2) some customers have
political/cultural reasons to prefer a centralized VCS with
repositories that can't be easily cloned.

I think that's enough for survival.  But it's not exciting, not sexy,
and not a recipe for drawing in new development talent.  Thus, if you try
to plan for big things, you will almost certainly fail because you won't
be able to collect the investment of developmen time required to
realize them.

What you *can* hope for is to ship occasional releases of high quality
and maintain Subversion's deserved reputation as the best of the pre-DVCS
version-control systems.

This is what I mean by setting realistic expectations.  It means gearing
down - accepting that your release tempo is going to be low and your
main goal is to keep the issue tracker relatively clean.

This is where I am now with GPSD.  I had to struggle a bit to accept it,
but the truth is GPSD is mature software that doesn't have much of
anything left to do in its application domain.  In an important way,
that is victory.

I'll pitch in here myself; I have plans to collect some more information
about the semantics of the dump format and add it to the documentation
already in the source tree.  Because I believe in finishing what I
started and leaving behind artifacts that say "Damn, that guy was a
pro."

You can still have that kind of excellence.  It's not a trivial thing.

svn_io_file_flush_to_disk fails on AFS filesystems on Darwin

[Quentin Smith]

Hi,

svn_io_file_flush_to_disk in subversion/libsvn_subr/io.c fails to flush 
files on AFS filesystems on Darwin. The user-visible experience is:


$ svn commit
svn: E25: Commit failed (details follow):
svn: E25: Can't write '/afs/path/db/txn-current' atomically
svn: E25: Can't flush file '/afs/path/db/svn-CNFY6N' to disk: Inappropriate 
ioctl for device

This is because Darwin defines F_FULLFSYNC, which on AFS (and maybe other 
filesystems?) returns ENOTTY.


The code ignores EINVAL with a comment about filesystems that don't 
support it; evidently on Darwin this also needs to include ENOTTY.


(Also, I suspect it should fall back from fcntl to fsync, since I suspect 
fsync /does/ work on AFS filesystems.)


I repro'd this with svn 1.10.3 (r1842928), but inspection of trunk 
confirms the code has not changed.


Cheers,
--Quentin