Re: [translations] Our language downloads are chaotic.

[David Kastrup]

Federico Bruni  writes:

> Il giorno gio 27 feb 2020 alle 16:17, David Kastrup  ha
> scritto:

>> The thing is a nightmare to diagnose since our web server settings 
>> tell
>> the server to serve xxx.it.extension in preference of xxx.extension if
>> your browser settings say that you prefer Italian.
>> So when a link in the Italian documentation demands xxx.extension, a
>> developer having Italian set as preference in their browser will see
>> xxx.it.extension and think things work well.  But there are two
>> problems
>> with that: if somebody without Italian in their preferences tries the
>> Italian language version, they get thrown back all the time at a
>> different language (English, or their own preferred language).  And if
>> they download a PDF file, it will be named notation.pdf but correspond
>> to notation.it.pdf.
>> 
>
> AFAICS your reasoning is correct for HTML links, but not for links to
> other types of files (such as PDF files).

Ok, yes.  The Apache documentation talks about html extensions.

> Take this page as example:
> http://lilypond.org/doc/v2.19/Documentation/web/notation.it.html
>
> The link to the notation PDF file is a direct link. Content
> negotiation doesn't play any role here.
> http://lilypond.org/doc/v2.19/Documentation/notation.pdf
>
> This means that we must fix the link.
> Catalan and spanish pages are the only languages which link to the
> translated PDF file:
> http://lilypond.org/doc/v2.19/Documentation/web/notation.ca.html links
> to notation.ca.pdf
> http://lilypond.org/doc/v2.19/Documentation/web/notation.es.html links
> to notation.es.pdf
>
>
> The link for content negotiation is:
> http://lilypond.org/doc/v2.19/Documentation/web/notation
>
> and will resolve to the HTML file of the browser language.
> There's no link for PDF file content negotiation, at least in this
> setup.

Yes.  And it would appear that the masslink scripts fill in nodes from
English that would otherwise be missing.  If they do that on the webpage
material as well (I really don't have a clue which of make/* and
stepmake/* is relevant for what), the fallback of a requested language
should always be to English.

>> So I think the proper thing to do would be to rely on automatic
>> language selection _only_ for the landing page(s) of the LilyPond web
>> site and use explicit language links for everything we do.
>> 
>
> I agree. This has been already proposed in issue 2273:
> https://sourceforge.net/p/testlilyissues/issues/2273/

Some time ago.

>> Which means that we'll likely need to generate/link xxx.en.extension
>> versions as well so that the English documentation, once selected,
>> manages to stick with English.
>> 
>
> I don't see the reason for using xxx.en.extension.
> If 2273 is fixed, localized pages will have xxx.LANG.html links while
> english pages will have simply xxx.html, as it happens with offline 
> website. There's no possible conflict.

I am not sure whether there will be no conflict if someone whose browser
settings specify his preference for Italian tries to read the English
documentation.  Wouldn't the server then replace the non-extension links
with Italian ones?  Or does that only happen for non-existing files?

>> There is the odd case of untranslated pages that should fall back to
>> browser defaulted substitution languages.  Those would be reasonable
>> candidates for getting called with non-explicit extensions.  But then
>> anything linked from within them would again be a candidate for using
>> non-explicit extensions.
>> 
>
> Untranslated pages are currently generated from english sources. So
> there's a file "duplication": two equal xxx.html and xxx.LANG.html
> files. No need to fallback then.

That has the disadvantage of somebody who has Czech as his first
language and German as his second in the browser.  Non-existing Czech
pages would then fall back straight to English.  But if they get an
"explicitly" German page, following any links in there would lead to
German documentation even if Czech were available.

I am not sure we can reliably do better.

-- 
David Kastrup

Re: [translations] Our language downloads are chaotic.

[Federico Bruni]

Il giorno gio 27 feb 2020 alle 16:17, David Kastrup  ha 
scritto:

Federico Bruni  writes:


 Il giorno gio 27 feb 2020 alle 02:10, David Kastrup  ha
 scritto:

 For example, try downloading any French documentation from
  
if

 your browser locale is not French.  Possibly even if it is.




 I am currently setting up the size listings to correspond with the
 actual files linked there, but the actual files apparently are
 incorrect
 all too often.
 I've not yet discovered a coherent strategy in the various
 translations
 how they attempt to, by and large, get a hold of downloads in their
 own
 language.
 The download details pages are where the sizes are currently given
 (the
 others are without size currently as far as I can tell: it should
 not be
 too much work to reorganise size indications once I am finished, to
 get
 more of them).



 I see that the link to the PDF file is correct in the spanish page:
 http://lilypond.org/doc/v2.19/Documentation/web/notation.es.html

 So it seems that the other languages should be fixed.
 I don't know where though.

 The files are present, for example:
 http://lilypond.org/doc/v2.19/Documentation/notation.it.pdf

 it's only a wrong URL.

 The links in the website generated by make doc are correct, as you 
can

 see if you open a locally built website.


The thing is a nightmare to diagnose since our web server settings 
tell

the server to serve xxx.it.extension in preference of xxx.extension if
your browser settings say that you prefer Italian.

So when a link in the Italian documentation demands xxx.extension, a
developer having Italian set as preference in their browser will see
xxx.it.extension and think things work well.  But there are two 
problems

with that: if somebody without Italian in their preferences tries the
Italian language version, they get thrown back all the time at a
different language (English, or their own preferred language).  And if
they download a PDF file, it will be named notation.pdf but correspond
to notation.it.pdf.



AFAICS your reasoning is correct for HTML links, but not for links to 
other types of files (such as PDF files).

Take this page as example:
http://lilypond.org/doc/v2.19/Documentation/web/notation.it.html

The link to the notation PDF file is a direct link. Content negotiation 
doesn't play any role here.

http://lilypond.org/doc/v2.19/Documentation/notation.pdf

This means that we must fix the link.
Catalan and spanish pages are the only languages which link to the 
translated PDF file:
http://lilypond.org/doc/v2.19/Documentation/web/notation.ca.html links 
to notation.ca.pdf
http://lilypond.org/doc/v2.19/Documentation/web/notation.es.html links 
to notation.es.pdf



The link for content negotiation is:
http://lilypond.org/doc/v2.19/Documentation/web/notation

and will resolve to the HTML file of the browser language.
There's no link for PDF file content negotiation, at least in this 
setup.



So I think the proper thing to do would be to rely on automatic 
language

selection _only_ for the landing page(s) of the LilyPond web site and
use explicit language links for everything we do.



I agree. This has been already proposed in issue 2273:
https://sourceforge.net/p/testlilyissues/issues/2273/


Which means that we'll likely need to generate/link xxx.en.extension
versions as well so that the English documentation, once selected,
manages to stick with English.



I don't see the reason for using xxx.en.extension.
If 2273 is fixed, localized pages will have xxx.LANG.html links while 
english pages will have simply xxx.html, as it happens with offline 
website. There's no possible conflict.




There is the odd case of untranslated pages that should fall back to
browser defaulted substitution languages.  Those would be reasonable
candidates for getting called with non-explicit extensions.

But then anything linked from within them would again be a candidate 
for

using non-explicit extensions.



Untranslated pages are currently generated from english sources. So 
there's a file "duplication": two equal xxx.html and xxx.LANG.html 
files. No need to fallback then.

Re: [translations] Our language downloads are chaotic.

[David Kastrup]

David Kastrup  writes:

> Han-Wen Nienhuys  writes:
>
>> On Thu, Feb 27, 2020 at 4:17 PM David Kastrup  wrote:
>>
>>> Does anybody have a better idea than that?
>>
>> couldn't we rename the localized downloads as
>>
>>   notation_de.pdf
>>
>> and then link to them explicitly?

I think linking explicitly should always work (except possibly in
English, see below).

> We have in Documentation/web/server/lilypond.org.htaccess
>
> # Add extensions for language negotiation
> # See: https://www.w3.org/International/questions/qa-apache-lang-neg
> AddLanguage ca .ca
> AddLanguage cs .cs
> AddLanguage de .de
> AddLanguage es .es
> AddLanguage fr .fr
> AddLanguage hu .hu
> AddLanguage it .it
> AddLanguage ja .ja
> AddLanguage nl .nl
> AddLanguage pt .pt
> AddLanguage zh .zh
>
> so it's really up to us what kind of language translation we do.  I
> wonder whether adding
>
> DefaultLanguage en
>
> would improve or deteriorate matters.  It's not quite clear to me what
> would happen then.

Well, I don't think Apache will have a chance of distinguishing "I am
asking without language-specific extension because I explicitly want
English" from "I am asking without language-specific extension because I
want language-based selection", no matter what settings it has.

And we need to have a sensible offline-browsable behavior as well.

So I think we won't get around an .en suffix at least in the server
trees (rather than the offline ones) in the course of solving this
problem.

-- 
David Kastrup

Re: [translations] Our language downloads are chaotic.

[David Kastrup]

Han-Wen Nienhuys  writes:

> On Thu, Feb 27, 2020 at 4:17 PM David Kastrup  wrote:
>
>> Does anybody have a better idea than that?
>
> couldn't we rename the localized downloads as
>
>   notation_de.pdf
>
> and then link to them explicitly?

We have in Documentation/web/server/lilypond.org.htaccess

# Add extensions for language negotiation
# See: https://www.w3.org/International/questions/qa-apache-lang-neg
AddLanguage ca .ca
AddLanguage cs .cs
AddLanguage de .de
AddLanguage es .es
AddLanguage fr .fr
AddLanguage hu .hu
AddLanguage it .it
AddLanguage ja .ja
AddLanguage nl .nl
AddLanguage pt .pt
AddLanguage zh .zh

so it's really up to us what kind of language translation we do.  I
wonder whether adding

DefaultLanguage en

would improve or deteriorate matters.  It's not quite clear to me what
would happen then.

-- 
David Kastrup

Re: [translations] Our language downloads are chaotic.

[Han-Wen Nienhuys]

On Thu, Feb 27, 2020 at 4:17 PM David Kastrup  wrote:

> Does anybody have a better idea than that?

couldn't we rename the localized downloads as

  notation_de.pdf

and then link to them explicitly?

-- 
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen

Re: [translations] Our language downloads are chaotic.

[David Kastrup]

Federico Bruni  writes:

> Il giorno gio 27 feb 2020 alle 02:10, David Kastrup  ha
> scritto:
>> For example, try downloading any French documentation from
>>  if
>> your browser locale is not French.  Possibly even if it is.
>> 
>
>> I am currently setting up the size listings to correspond with the
>> actual files linked there, but the actual files apparently are
>> incorrect
>> all too often.
>> I've not yet discovered a coherent strategy in the various 
>> translations
>> how they attempt to, by and large, get a hold of downloads in their
>> own
>> language.
>> The download details pages are where the sizes are currently given 
>> (the
>> others are without size currently as far as I can tell: it should
>> not be
>> too much work to reorganise size indications once I am finished, to
>> get
>> more of them).
>> 
>
> I see that the link to the PDF file is correct in the spanish page:
> http://lilypond.org/doc/v2.19/Documentation/web/notation.es.html
>
> So it seems that the other languages should be fixed.
> I don't know where though.
>
> The files are present, for example:
> http://lilypond.org/doc/v2.19/Documentation/notation.it.pdf
>
> it's only a wrong URL.
>
> The links in the website generated by make doc are correct, as you can
> see if you open a locally built website.

The thing is a nightmare to diagnose since our web server settings tell
the server to serve xxx.it.extension in preference of xxx.extension if
your browser settings say that you prefer Italian.

So when a link in the Italian documentation demands xxx.extension, a
developer having Italian set as preference in their browser will see
xxx.it.extension and think things work well.  But there are two problems
with that: if somebody without Italian in their preferences tries the
Italian language version, they get thrown back all the time at a
different language (English, or their own preferred language).  And if
they download a PDF file, it will be named notation.pdf but correspond
to notation.it.pdf.

So I think the proper thing to do would be to rely on automatic language
selection _only_ for the landing page(s) of the LilyPond web site and
use explicit language links for everything we do.

Which means that we'll likely need to generate/link xxx.en.extension
versions as well so that the English documentation, once selected,
manages to stick with English.

There is the odd case of untranslated pages that should fall back to
browser defaulted substitution languages.  Those would be reasonable
candidates for getting called with non-explicit extensions.

But then anything linked from within them would again be a candidate for
using non-explicit extensions.

So to make this work in a best-of-all scenario, we'd need
language-specific subdirectories where all languages are present in each
language-specific subdirectory, but with the dedicated language only
linking to files with its dedicated extension and all other languages
linking to files with the non-dedicated name.

This.  is.  a.  nightmare.

If we don't want to drown in duplication, I think we'll keep with one
directory but explicit links.  That means that if you get a German
substitution page for a missing Czech page and use links within it, they
will always lead to German pages even if a corresponding Czech page
would be available.

Or we have two top-level directories: one driven by browser preference,
one hardlocked to selected languages, and using some
"give-me-language-x" link anywhere in the documentation leads you to the
hardlocked variant which cannot heed browser preferences.  Probably just
a matter of a different .htaccess file?

But for now, I think the "stick-with-one-language-if-node-exists"
version best.  It will mean that when new translated nodes appear, links
to them will need to get (manually?) changed from "generic" to
"language-specific".

And we'll want .en versions of all files that are hardwired to stick in
the .en universe, or English will be the least persistent language of
all.

Does anybody have a better idea than that?

-- 
David Kastrup

Re: [translations] Our language downloads are chaotic.

[Federico Bruni]

Il giorno gio 27 feb 2020 alle 02:10, David Kastrup  ha 
scritto:


For example, try downloading any French documentation from
 if
your browser locale is not French.  Possibly even if it is.




I am currently setting up the size listings to correspond with the
actual files linked there, but the actual files apparently are 
incorrect

all too often.

I've not yet discovered a coherent strategy in the various 
translations
how they attempt to, by and large, get a hold of downloads in their 
own

language.

The download details pages are where the sizes are currently given 
(the
others are without size currently as far as I can tell: it should not 
be
too much work to reorganise size indications once I am finished, to 
get

more of them).



I see that the link to the PDF file is correct in the spanish page:
http://lilypond.org/doc/v2.19/Documentation/web/notation.es.html

So it seems that the other languages should be fixed.
I don't know where though.

The files are present, for example:
http://lilypond.org/doc/v2.19/Documentation/notation.it.pdf

it's only a wrong URL.

The links in the website generated by make doc are correct, as you can 
see if you open a locally built website.