Bug#924172: www.debian.org: differences under english/ between builds in stretch and buster

[Laura Arjona Reina]

Hi all
Sorry for the long delay to come back to this.

I have tested the workaround proposed by Cyril:

https://salsa.debian.org/webmaster-team/webwml/-/tree/pu/partial-workaround-924172

and arrived to the same conclusions:

* It makes no difference (and thus no harm) when applied in stretch
* In buster, it builds correctly the sitemaps, with no references to
"../english" paths (I tested Spanish and English, and the Spanish
translation seemed also to be complete).

OTOH, now that wml 2.28.0 is packaged in testing (thanks Axel and
Shlomi!), I have installed wml and slice from testing in my buster box,
and tried to build, and found that sitemap.en.html cannot be built (See
error below. But just for the curious, because I think our way to go is
to integrate Cyril's workaround, and with the patch, the files build
correctly with wml 2.28.0).

So I think it's good to accept it and unblock this way the upgrade to
buster of www-master. Later we can file bugs for the not-complete
translations.

Kind regards,
-- 
Laura Arjona Reina
https://wiki.debian.org/LauraArjona


--- Error when building sitemap.en.html in a buster machine, but with
wml and slice 2.28.0 (from testing) -- Just for the record, because
we'll use another workaround



wml -q -D CUR_YEAR=2020 -o UNDEFuEN:sitemap.en.html@g+w   -D
CUR_LANG=English -D CUR_ISO_LANG=en -D CUR_LOCALE=en_US.UTF-8 -D
CHARSET=utf-8 \
  ../english/sitemap.wml
ePerl:Error: Perl runtime error (interpreter rc=2)

 Contents of STDERR channel: -
couldn't open ./../index.wml or
/home/larjona/Documentos/debian/www/webwml/english/../index.wml: No such
file or directory
Died at /tmp/0uDhaUPMnJ/wml.tmp1 line 750.
--
** WML:Break: Error in Pass 3 (rc=1).
Died at /usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 402.

TheWML::Frontends::Wml::Runner::_run_pass(TheWML::Frontends::Wml::Runner=HASH(0x563974ead458),
3, SCALAR(0x56397544aaf0), REF(0x56397544aac0), REF(0x56397544aad8))
called at /usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 440

TheWML::Frontends::Wml::Runner::_passes_loop(TheWML::Frontends::Wml::Runner=HASH(0x563974ead458))
called at /usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 727

TheWML::Frontends::Wml::Runner::_output_and_cleanup(TheWML::Frontends::Wml::Runner=HASH(0x563974ead458))
called at /usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 932

TheWML::Frontends::Wml::Runner::run_with_ARGV(TheWML::Frontends::Wml::Runner=HASH(0x563974ead458),
HASH(0x56397543aed0)) called at /usr/bin/wml line 47
make: *** [Makefile:56: sitemap.en.html] Error 2

Bug#924172: www.debian.org: differences under english/ between builds in stretch and buster

[Laura Arjona Reina]

El 10/8/20 a las 22:47, Laura Arjona Reina escribió:

> 
> So I think it's good to accept it and unblock this way the upgrade to
> buster of www-master. Later we can file bugs for the not-complete
> translations.
> 

This has been done in MR #524:

https://salsa.debian.org/webmaster-team/webwml/-/merge_requests/524

(main commit 83f7cae0178074e5e4f913168b4b42c7be90af13)

Kind regards,
-- 
Laura Arjona Reina
https://wiki.debian.org/LauraArjona

Bug#924172: www.debian.org: differences under english/ between builds in stretch and buster

[Shlomi Fish]

Hi Axel!

You can find my attempt to fix the debian build here:

https://www.shlomifish.org/Files/files/code/wml.debian-shlomify-1.diff

The sed addition to debian/rules seems to improve matters, but the restoration
of p9_slice (to fix loading SliceTermParser.pm ) gives dpkg-source a fit. I
really find the debian gbp stuff puzzling.

here is the script i used:

https://www.shlomifish.org/Files/files/text/docker-wml-debian-perl-v0.1.0.txt

On Thu, 11 Jun 2020 17:00:42 +0200
Axel Beckert  wrote:

> Hi Shlomi,
> 
> [taking Cyril and Laura out of the loop]
> 
> Shlomi Fish wrote:
> > > IIRC I had test suite failures. Need to check 2.28 again. (And yes, I
> > > know, I haven't filed an upstream bug report yet — still haven't
> > > figured out what exactly is the cause for the failures.)  
> > 
> > Ah - thanks for the update. I guess I can try building the package
> > myself in a VM/container. For the record, the build and tests pass
> > fine in travis-ci/ubuntu bionic , locally on fedora and mageia, on
> > the mageia build system, and on appveyor/mswin10/cygwin .  
> 
> Yeah, another reason for not having it reported upstream is that I'm
> not yet sure if one of the Debian patches is the culprit.

It seems like this is the case.

> 
> Regarding Travis CI: IMHO its value is quite limited by the fact that
> Travis is nearly always two or more years behind the latest Ubuntu LTS
> release. (Actually I'm kinda surprised that they in the meanwhile
> managed to support 18.04. The last time I looked (IIRC a few months
> ago around the 20.04 release) they were still on 16.04.
> 

Yes, you can run docker containers in travis.

> > > Will do — as soon I get it building again.  
> > 
> > I'll try to help.  
> 
> Appreciated, but I would kinda feel bad in case one of the Debian
> patches will be identified as the culprit and someone else had to do
> my work. :-)

No worries.

> 
> > > > Converting some of my sites away from wml has shortened their
> > > > build times considerably.  
> > > 
> > > But are they as flexible as before while still being statically
> > > compiled? I doubt.  
> > 
> > They are still "statically compiled" (or generated:
> > https://github.com/shlomif/shlomif-tech-diary/blob/master/static-site-generators--despair.md
> > ), and I verified that they generated the same output before and
> > after using "diff -u -r", html-minifier and other tools. Template
> > Toolkit is fairly flexible [...]  
> 
> Ah, TT. Oh well, from my point of view, TT always felt too generic to
> me. I had the feeling that I'd have to write a lot of the
> infrastructure for static compiling that WML already offers (mainly
> wmk, but also some of the passes like splice), myself. So I never
> considered it to be a WML replacement but rather something that is on
> the same level as e.g. Embperl.)
>

Perhaps.
 
>   Regards, Axel



-- 

Shlomi Fish   https://www.shlomifish.org/
UNIX Fortune Cookies - https://www.shlomifish.org/humour/fortunes/

Fortran - there isn’t a way to do it... oh wait! Now there is.
— https://www.shlomifish.org/humour/ways_to_do_it.html

Please reply to list if it's a mailing list post - https://shlom.in/reply .

Bug#924172: www.debian.org: differences under english/ between builds in stretch and buster

[Axel Beckert]

Hi Shlomi,

[taking Cyril and Laura out of the loop]

Shlomi Fish wrote:
> > IIRC I had test suite failures. Need to check 2.28 again. (And yes, I
> > know, I haven't filed an upstream bug report yet — still haven't
> > figured out what exactly is the cause for the failures.)
> 
> Ah - thanks for the update. I guess I can try building the package
> myself in a VM/container. For the record, the build and tests pass
> fine in travis-ci/ubuntu bionic , locally on fedora and mageia, on
> the mageia build system, and on appveyor/mswin10/cygwin .

Yeah, another reason for not having it reported upstream is that I'm
not yet sure if one of the Debian patches is the culprit.

Regarding Travis CI: IMHO its value is quite limited by the fact that
Travis is nearly always two or more years behind the latest Ubuntu LTS
release. (Actually I'm kinda surprised that they in the meanwhile
managed to support 18.04. The last time I looked (IIRC a few months
ago around the 20.04 release) they were still on 16.04.

> > Will do — as soon I get it building again.
> 
> I'll try to help.

Appreciated, but I would kinda feel bad in case one of the Debian
patches will be identified as the culprit and someone else had to do
my work. :-)

> > > Converting some of my sites away from wml has shortened their
> > > build times considerably.
> > 
> > But are they as flexible as before while still being statically
> > compiled? I doubt.
> 
> They are still "statically compiled" (or generated:
> https://github.com/shlomif/shlomif-tech-diary/blob/master/static-site-generators--despair.md
> ), and I verified that they generated the same output before and
> after using "diff -u -r", html-minifier and other tools. Template
> Toolkit is fairly flexible [...]

Ah, TT. Oh well, from my point of view, TT always felt too generic to
me. I had the feeling that I'd have to write a lot of the
infrastructure for static compiling that WML already offers (mainly
wmk, but also some of the passes like splice), myself. So I never
considered it to be a WML replacement but rather something that is on
the same level as e.g. Embperl.)

Regards, Axel
-- 
 ,''`.  |  Axel Beckert , https://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5
  `-|  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE

Bug#924172: www.debian.org: differences under english/ between builds in stretch and buster

[Shlomi Fish]

Hi Axel,

On Thu, 11 Jun 2020 13:51:28 +0200
Axel Beckert  wrote:

> Hi Shlomi,
> 
> Shlomi Fish wrote:
> > 2. The newest release of wml is 2.28.0 (see
> > https://github.com/thewml/website-meta-language/releases )  
> 
> I know.
> 
> > while Debian sid is stuck at 2.12.x (see
> > https://packages.debian.org/sid/wml ).  
> 
> Yes, as I didn't get any newer releases (up to 2.24) to build properly
> anymore. You can see my tries here:
> https://salsa.debian.org/debian/wml/-/tree/2.20.4-pkg-incomplete
> (ignore the version in the branch name, I didn't expect this to last
> several upstream releases).
> 
> IIRC I had test suite failures. Need to check 2.28 again. (And yes, I
> know, I haven't filed an upstream bug report yet — still haven't
> figured out what exactly is the cause for the failures.)
> 

Ah - thanks for the update. I guess I can try building the package myself in a
VM/container. For the record, the build and tests pass fine in travis-ci/ubuntu
bionic , locally on fedora and mageia, on the mageia build system, and on
appveyor/mswin10/cygwin .

> > I'd rather not support such an old release,  
> 
> It's not about Debian Unstable, it's about the WML version in current
> Debian Unstable (which only happens to be the same upstream version as
> in Debian Unstable for the reasons mentioned above) and Debian (well,
> I) will support this until the EoL of Debian 10 Buster.
>

ok.
 
> > so if the new version still exhibits some
> > regressions, please send a failing testcase to
> > https://github.com/thewml/website-meta-language/tree/master/src/wml_test and
> > I'll try to fix it.  
> 
> Will do — as soon I get it building again.
> 

I'll try to help.

> > Converting some of my sites away from wml has shortened their build times
> > considerably.  
> 
> But are they as flexible as before while still being statically
> compiled? I doubt.
> 

They are still "statically compiled" (or generated:
https://github.com/shlomif/shlomif-tech-diary/blob/master/static-site-generators--despair.md
), and I verified that they generated the same output before and after using
"diff -u -r", html-minifier and other tools. Template Toolkit is fairly
flexible and has fewer "WTF?" surprises than wml, as I discovered some
misrendered output during the conversions. There may be some features that only
wml has and TT doesn't, but I think the benefits outweigh the drawbacks.

>   Regards, Axel



-- 

Shlomi Fish   https://www.shlomifish.org/

Objective Visual Turbo Global jIronOpenPerl++.NET™ Enterprise Edition♭
Professional Home Premium Ultimate 64-bit Single-user.
— based on a Freenode #perl conversation ( https://is.gd/cCUBY2 )

Please reply to list if it's a mailing list post - https://shlom.in/reply .

Bug#924172: www.debian.org: differences under english/ between builds in stretch and buster

[Axel Beckert]

Hi again,

Axel Beckert wrote:
> > I'd rather not support such an old release,
> 
> It's not about Debian Unstable, it's about the WML version in current
> Debian Unstable (which only happens to be the same upstream version as
 

of course this should have been "Debian Stable", not "Debian Unstable again.

> in Debian Unstable for the reasons mentioned above) and Debian (well,
> I) will support this until the EoL of Debian 10 Buster.

Regards, Axel
-- 
 ,''`.  |  Axel Beckert , https://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5
  `-|  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE

Bug#924172: www.debian.org: differences under english/ between builds in stretch and buster

[Axel Beckert]

Hi Shlomi,

Shlomi Fish wrote:
> 2. The newest release of wml is 2.28.0 (see
> https://github.com/thewml/website-meta-language/releases )

I know.

> while Debian sid is stuck at 2.12.x (see
> https://packages.debian.org/sid/wml ).

Yes, as I didn't get any newer releases (up to 2.24) to build properly
anymore. You can see my tries here:
https://salsa.debian.org/debian/wml/-/tree/2.20.4-pkg-incomplete
(ignore the version in the branch name, I didn't expect this to last
several upstream releases).

IIRC I had test suite failures. Need to check 2.28 again. (And yes, I
know, I haven't filed an upstream bug report yet — still haven't
figured out what exactly is the cause for the failures.)

> I'd rather not support such an old release,

It's not about Debian Unstable, it's about the WML version in current
Debian Unstable (which only happens to be the same upstream version as
in Debian Unstable for the reasons mentioned above) and Debian (well,
I) will support this until the EoL of Debian 10 Buster.

> so if the new version still exhibits some
> regressions, please send a failing testcase to
> https://github.com/thewml/website-meta-language/tree/master/src/wml_test and
> I'll try to fix it.

Will do — as soon I get it building again.

> Converting some of my sites away from wml has shortened their build times
> considerably.

But are they as flexible as before while still being statically
compiled? I doubt.

Regards, Axel
-- 
 ,''`.  |  Axel Beckert , https://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5
  `-|  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE

Bug#924172: www.debian.org: differences under english/ between builds in stretch and buster

[Shlomi Fish]

Hi all,

On Mon, 8 Jun 2020 00:35:26 +0200
Cyril Brulebois  wrote:

> (Looping in wml package maintainer + original author)
> 

Some notes:

1. I am not the original author of wml, just the current upstream maintainer
(though I believe that I have invested quite a lot of work in it).

2. The newest release of wml is 2.28.0 (see
https://github.com/thewml/website-meta-language/releases ) while Debian sid is
stuck at 2.12.x (see https://packages.debian.org/sid/wml ). I'd rather not
support such an old release, so if the new version still exhibits some
regressions, please send a failing testcase to
https://github.com/thewml/website-meta-language/tree/master/src/wml_test and
I'll try to fix it.

For a joke about it, see:
https://www.shlomifish.org/humour/fortunes/tinic.html#hamakor-discs-mozilla-1.1-1

3. I recommend the Debian site to gradually transition away from using wml,
because while wml is fairly powerful and flexible, it is slow, quirky and hard
to learn. It is a hindrance that the debian site takes 3 hours to build on a
modern computer:

https://xkcd.com/303/

Converting some of my sites away from wml has shortened their build times
considerably.

Regards,

Shlomi

> Hi!
> 
> Laura Arjona Reina  (2020-06-07):
> > Thanks Cyril for your work here.
> > 
> > We're trying to fix all the issues so the migration of www-master to
> > buster can be done as soon as possible.  
> 
> You're welcome, helping towards that goal was my original intent.
> 
> > I'll comment on the issues you found:
> > 
> > 1. Changes related to canonicalization(?) of the URL
> > 
> > The files affected are:
> > 
> > 1.1.- */News/news.*.rdf  (all languages)
> > 
> > All those files are built using /english/News/news.rdf.in which includes
> > this line:
> > 
> > 
> > 
> > it seems that the variable $(HOME) is interpreted differently by the
> > make or wml commands in buster.
> > 
> > I have played with the "-D HOME~." line in /english/.wmlrc but it does
> > not solve the issue.
> > 
> > The file is built (in english and all the other languages except
> > Chinese) by lines 36-37 of the /english/News/Makefile:
> > 
> > $(WML) $(shell egrep '^-D (CUR_|CHAR)' ../.wmlrc) \
> > $(ENGLISHDIR)/News/news.rdf.in
> > 
> > If I change that to
> > 
> > $(WML) $(shell egrep '^-D (CUR_|CHAR)' ../.wmlrc) news.rdf.in
> > 
> > then the English file is built with correct URL to the CSS, but the
> > other languages fail.
> > 
> > I don't know how to solve this, except using an absolute reference to
> > the CSS file in the /english/News/news.rdf.in instead of the $(HOME)
> > variable.  
> 
> I'll skip this for now, to concentrate on the topic I've debugged this
> evening; this might be related though!
> 
> > 1.2. */sitemap.*.html  (all languages)
> > 
> > I guess it's the same problem (that the $(HOME) variable is used and
> > interpreted wrongly with the new make or wml), but this file and how it
> > is built is more cryptic for me so I wouldn't know how to start.
> > 
> > The sitemap would be completely broken until this issue is fixed, if
> > we migrate www-master to buster :/
> > 
> > Help needed!  
> 
> For the record, we're talking about:
>   https://salsa.debian.org/webmaster-team/webwml
> 
> The problem can be triggered with:
>   make -C english sitemap.en.html
> 
> Another file in the same directory can be obtained with:
>   make -C english index.en.html
> 
> 
> OK, so I've spent some time on this. Based on the HOME mention, I tried
> to switch it from `-D HOME~.` to `-D HOME=.` in english/.wmlrc but that
> wasn't sufficient. I had to turn *all* such defines in that file from
> `~` to `=` to get a reasonable sitemap.
> 
> The wml_intro.7 manpage quickly explains the difference between both,
> but that seems to be happening in passes 2 and 3; english/sitemap.wml is
> one of the only files were pass protection is used, but that turned out
> to be a red herring…
> 
> 
> Adding the `-v2` flag to the `$(WML)` calls in english/Makefile made the
> issue apparent: as early as pass 1, we have different parameters being
> sent to wml_p1_ipp between stretch (2.0.12) and buster (2.12.2), among
> which:
> 
>  * stretch:
> 
> "-DBUGS=Bugs"
> "-DINTRO=intro"
> 
>  * buster:
> "-DBUGS=../english/Bugs"
> "-DINTRO=../english/intro" 
> 
> for our english/sitemap.wml
> 
> BUT
> 
> that's not the case for index.en.html, built from english/index.wml!
> 
> At this point, I suspected the origin of the change in behavior was the
> path to the input file:
> 
>   $(WML) … index.wml is OK
>   $(WML) … ../english/sitemap.wml is not OK
> 
> because the latter triggers the insertion of `../english/` in lots of
> places (defined with `~` instead of `=`) with 2.12.2, which was not the
> case with 2.0.12.
> 
> 
> Dropping the ../english/ prefix for sitemap.wml leads to a reasonable
> diff for the English sitemap (sitemap.en.html):
> 
> --- stretch.en.html   2020-06-08 00:07:43.916768223 +0200
> +++ buster.en.html  

Bug#924172: www.debian.org: differences under english/ between builds in stretch and buster

[Cyril Brulebois]

Cyril Brulebois  (2020-06-08):
> Dropping the ../english/ prefix for sitemap.wml leads to a reasonable
> diff for the English sitemap (sitemap.en.html):
> 
> --- stretch.en.html   2020-06-08 00:07:43.916768223 +0200
> +++ buster.en.html2020-06-08 00:07:47.248781809 +0200
> @@ -4,8 +4,8 @@
>
>Site map for Debian web pages 
>mailto:webmas...@debian.org";>
> -  
> -  
> +  
> +  
>
>
>
> @@ -377,7 +377,7 @@
>  
>  Last Modified: Sun, Jun 7 21:49:47 UTC 2020
>   
> -Last Built: Sun, Jun 7 22:07:43 UTC 2020
> +Last Built: Sun, Jun 7 22:07:46 UTC 2020
>
>Copyright © 1997-2020
>   https://www.spi-inc.org/";>SPI and others; See  href="./license" rel="copyright">license terms
> 
> Therefore, I suspect we *might* be able to get away with this if we were
> to add a symlink from all language directories, from $LANG/sitemap.wml
> to ../english/sitemap.wml, as a companion change to dropping the
> ../english prefix mentioned above (only tested successfully on English).
> I'll be checking this hypothesis and possibly pushing a branch / opening
> an MR if that works fine.

OK. This isn't so simple. I've pushed this branch:
  
https://salsa.debian.org/webmaster-team/webwml/-/tree/pu/partial-workaround-924172

I'm attaching several sitemap files, built either in stretch or in
buster, in English or in French, from the master branch or from the
workaround branch:

buster.en.html.master
buster.en.html.workaround
buster.fr.html.master
buster.fr.html.workaround
stretch.en.html.master
stretch.en.html.workaround
stretch.fr.html.master
stretch.fr.html.workaround

For each language, diffing the stretch built between master and the
workaround yields no (practical) differences.

Diffing stretch.en.master.html and buster.en.master.html shows many
../english/ being added; which could be arguably seen as a no-op.

Diffing stretch.fr.master.html and buster.fr.master.html shows many
../english/ being added; which breaks navigation as that's another
language…

Diffing stretch.en.master.html and buster.en.workaround.html shows
no practical differences (per my previous mail).

Diffing stretch.fr.master.html and buster.fr.workaround.html shows
the lost translations between (patched or unpatched) French version
built in stretch and patched French version built in English. It
seems we're losing translations for items which would have suffered
from the ../english/ addition.

Interestingly, this list seems to match the special cases defined in
english/sitemap.wml (which is about pass protection this time, see
the escape tag definition and comment above). I haven't tried to
double check the flags passed to each pass in each cases though
(running out of time).


Overall, it seems my proposed patch papers around the issue but not
fully. But it could help unstuck the move to buster if we agree the
partial translations for the sitemaps are better than rather broken
sitemaps.


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#924172: www.debian.org: differences under english/ between builds in stretch and buster

[Cyril Brulebois]

(Looping in wml package maintainer + original author)

Hi!

Laura Arjona Reina  (2020-06-07):
> Thanks Cyril for your work here.
> 
> We're trying to fix all the issues so the migration of www-master to
> buster can be done as soon as possible.

You're welcome, helping towards that goal was my original intent.

> I'll comment on the issues you found:
> 
> 1. Changes related to canonicalization(?) of the URL
> 
> The files affected are:
> 
> 1.1.- */News/news.*.rdf  (all languages)
> 
> All those files are built using /english/News/news.rdf.in which includes
> this line:
> 
> 
> 
> it seems that the variable $(HOME) is interpreted differently by the
> make or wml commands in buster.
> 
> I have played with the "-D HOME~." line in /english/.wmlrc but it does
> not solve the issue.
> 
> The file is built (in english and all the other languages except
> Chinese) by lines 36-37 of the /english/News/Makefile:
> 
>   $(WML) $(shell egrep '^-D (CUR_|CHAR)' ../.wmlrc) \
>   $(ENGLISHDIR)/News/news.rdf.in
> 
> If I change that to
> 
>   $(WML) $(shell egrep '^-D (CUR_|CHAR)' ../.wmlrc) news.rdf.in
> 
> then the English file is built with correct URL to the CSS, but the
> other languages fail.
> 
> I don't know how to solve this, except using an absolute reference to
> the CSS file in the /english/News/news.rdf.in instead of the $(HOME)
> variable.

I'll skip this for now, to concentrate on the topic I've debugged this
evening; this might be related though!

> 1.2. */sitemap.*.html  (all languages)
> 
> I guess it's the same problem (that the $(HOME) variable is used and
> interpreted wrongly with the new make or wml), but this file and how it
> is built is more cryptic for me so I wouldn't know how to start.
> 
> The sitemap would be completely broken until this issue is fixed, if
> we migrate www-master to buster :/
> 
> Help needed!

For the record, we're talking about:
  https://salsa.debian.org/webmaster-team/webwml

The problem can be triggered with:
  make -C english sitemap.en.html

Another file in the same directory can be obtained with:
  make -C english index.en.html


OK, so I've spent some time on this. Based on the HOME mention, I tried
to switch it from `-D HOME~.` to `-D HOME=.` in english/.wmlrc but that
wasn't sufficient. I had to turn *all* such defines in that file from
`~` to `=` to get a reasonable sitemap.

The wml_intro.7 manpage quickly explains the difference between both,
but that seems to be happening in passes 2 and 3; english/sitemap.wml is
one of the only files were pass protection is used, but that turned out
to be a red herring…


Adding the `-v2` flag to the `$(WML)` calls in english/Makefile made the
issue apparent: as early as pass 1, we have different parameters being
sent to wml_p1_ipp between stretch (2.0.12) and buster (2.12.2), among
which:

 * stretch:

"-DBUGS=Bugs"
"-DINTRO=intro"

 * buster:
"-DBUGS=../english/Bugs"
"-DINTRO=../english/intro" 

for our english/sitemap.wml

BUT

that's not the case for index.en.html, built from english/index.wml!

At this point, I suspected the origin of the change in behavior was the
path to the input file:

  $(WML) … index.wml is OK
  $(WML) … ../english/sitemap.wml is not OK

because the latter triggers the insertion of `../english/` in lots of
places (defined with `~` instead of `=`) with 2.12.2, which was not the
case with 2.0.12.


Dropping the ../english/ prefix for sitemap.wml leads to a reasonable
diff for the English sitemap (sitemap.en.html):

--- stretch.en.html 2020-06-08 00:07:43.916768223 +0200
+++ buster.en.html  2020-06-08 00:07:47.248781809 +0200
@@ -4,8 +4,8 @@
   
   Site map for Debian web pages 
   mailto:webmas...@debian.org";>
-  
-  
+  
+  
   
   
   
@@ -377,7 +377,7 @@
 
 Last Modified: Sun, Jun 7 21:49:47 UTC 2020
  
-Last Built: Sun, Jun 7 22:07:43 UTC 2020
+Last Built: Sun, Jun 7 22:07:46 UTC 2020
   
   Copyright © 1997-2020
  https://www.spi-inc.org/";>SPI and others; See license terms

Therefore, I suspect we *might* be able to get away with this if we were
to add a symlink from all language directories, from $LANG/sitemap.wml
to ../english/sitemap.wml, as a companion change to dropping the
../english prefix mentioned above (only tested successfully on English).
I'll be checking this hypothesis and possibly pushing a branch / opening
an MR if that works fine.

I'm sending this mail to reach out to upstream so they can comment on
whether the change was intentional or an unwanted side effect.

I might look at the News/RDF thing in the meanwhile.


> 3. Changes in a log file
> 
> Not important (and probably we shouldn't provide the card there, maybe
> in the debian-flyers repo?)
> 
> 4. Changes in ordering of coordinators
> 5. Changes in ordering under wnpp
> 6. Changes in order under l10n
> 8. More ordering changes (architectures, DSAs)
> 
> Thanks for reporting, and fo

Bug#924172: www.debian.org: differences under english/ between builds in stretch and buster

[Laura Arjona Reina]

Hello all,

Thanks Cyril for your work here.
We're trying to fix all the issues so the migration of www-master to
buster can be done as soon as possible.

I'll comment on the issues you found:

1. Changes related to canonicalization(?) of the URL

The files affected are:

1.1.- */News/news.*.rdf  (all languages)

All those files are built using /english/News/news.rdf.in which includes
this line:



it seems that the variable $(HOME) is interpreted differently by the
make or wml commands in buster.

I have played with the "-D HOME~." line in /english/.wmlrc but it does
not solve the issue.

The file is built (in english and all the other languages except
Chinese) by lines 36-37 of the /english/News/Makefile:

$(WML) $(shell egrep '^-D (CUR_|CHAR)' ../.wmlrc) \
$(ENGLISHDIR)/News/news.rdf.in

If I change that to

$(WML) $(shell egrep '^-D (CUR_|CHAR)' ../.wmlrc) news.rdf.in

then the English file is built with correct URL to the CSS, but the
other languages fail.

I don't know how to solve this, except using an absolute reference to
the CSS file in the /english/News/news.rdf.in instead of the $(HOME)
variable.

1.2. */sitemap.*.html  (all languages)

I guess it's the same problem (that the $(HOME) variable is used and
interpreted wrongly with the new make or wml), but this file and how it
is built is more cryptic for me so I wouldn't know how to start.

The sitemap would be completely broken until this issue is fixed, if we
migrate www-master to buster :/

Help needed!

2. Changes in mail address representation

I confirm this happens but it's indeed random changes in the addresses
obfuscation, so I don't consider this an issue.

3. Changes in a log file

Not important (and probably we shouldn't provide the card there, maybe
in the debian-flyers repo?)

4. Changes in ordering of coordinators
5. Changes in ordering under wnpp
6. Changes in order under l10n
8. More ordering changes (architectures, DSAs)

Thanks for reporting, and for the work towards reproducibility.
I think these are not blockers for the migration to buster of
www-master.debian.org

Maybe we could open a specific bug about these "reproducibility issues"
and see if somebody is willing/able to work on it?

7. Changes related to image width/height attributes

This is indeed as Julien commented, a bug in wml which is fixed in buster.

Thanks
-- 
Laura Arjona Reina
https://wiki.debian.org/LauraArjona

Bug#924172: www.debian.org: differences under english/ between builds in stretch and buster

[Julien Cristau]

On Sun, Mar 10, 2019 at 02:17:50AM +0100, Cyril Brulebois wrote:
> 7. Changes related to image width/height attributes
> ===
> 
> 
> --- stretch/webwml/english/partners/2018/index.en.html
> +++ buster/webwml/english/partners/2018/index.en.html
> 
> -   class="partnerlogo" width=height=>
> +   class="partnerlogo">
> 
> --- stretch/webwml/english/partners/2019/index.en.html
> +++ buster/webwml/english/partners/2019/index.en.html
> 
> -   class="partnerlogo" width=height=>
> +   class="partnerlogo">
> 
> -   class="partnerlogo" width=height=>
> +   class="partnerlogo">
> 
> --- stretch/webwml/english/partners/index.en.html
> +++ buster/webwml/english/partners/index.en.html
> 
> -   class="partnerlogo" width=height=>
> +   class="partnerlogo">
> 
> -   class="partnerlogo" width=height=>
> +   class="partnerlogo">
> 
> For some reason, width/height were empty, and they seem to be stripped
> now? Progress, maybe, except that might be hiding the root issue?
> 
This change is from https://github.com/thewml/website-meta-language/pull/1

Cheers,
Julien

Bug#924172: www.debian.org: differences under english/ between builds in stretch and buster

[Cyril Brulebois]

Package: www.debian.org
Severity: normal
User: www.debian@packages.debian.org
Usertags: scripts

Hi,

While working on rebuilding the website within stretch and buster, I've
noticed the following.

This is a curated summary based on diffing a stretch build tree and a
buster build tree, after building the english subdirectory only, with
USE_SAMPLE_FILES=1. This was built by diffing both directory, excluding
all files with 8 changes (assuming those were due to the webwml version
and dates of last modification/build changes, which a cursory look
confirms on dozens of them, but that really should be automated).

I'd think items #1 and #7 should be given high priority.


1. Changes related to canonicalization(?) of the URL


--- stretch/webwml/english/News/news.en.rdf
+++ buster/webwml/english/News/news.en.rdf
@@ -1,5 +1,5 @@
 
-
+
 http://www.w3.org/1999/02/22-rdf-syntax-ns#";
   xmlns="http://purl.org/rss/1.0/";

--- stretch/webwml/english/sitemap.en.html
+++ buster/webwml/english/sitemap.en.html

-
-  
-
+
+  
+

-  
+  

-   About Debian
-   Getting Debian
-   Support
-   Developers' Corner
+   About Debian
+   Getting Debian
+   Support
+   Developers' Corner

[ and many more ]


2. Changes in mail address representation
=

--- stretch/webwml/english/consultants/index.en.html
+++ buster/webwml/english/consultants/index.en.html

Many changes there, but I suppose this is due to some random obfuscation
of mail address, which is likely not reproducible anyway.


3. Changes in a log file


--- stretch/webwml/english/devel/misc/card.log
+++ buster/webwml/english/devel/misc/card.log

No surprises here, with versions for the build chain being bumped, and
timestamps.


4. Changes in ordering of coordinators
==

--- stretch/webwml/english/devel/website/translation_coordinators.en.html
+++ buster/webwml/english/devel/website/translation_coordinators.en.html

-Arabic: Ayman Negm , Mohammed Adnène 
Trojette 
+Arabic: Mohammed Adnène Trojette , 
Ayman Negm 

[ Ditto for: Arabic, Chinese, Indonesian, Persian, Polish, Romanian ]

Is it expected to have a random order? Or should this be made to build
and list people in a reproducible order?


5. Changes in ordering under wnpp
=

--- stretch/webwml/english/devel/wnpp/*
+++ buster/webwml/english/devel/wnpp/*

Should packages be listed in a reproducible order?


6. Changes in order under l10n
==

--- stretch/webwml/english/international/l10n/po-debconf/rank.en.html
+++ buster/webwml/english/international/l10n/po-debconf/rank.en.html

--- stretch/webwml/english/international/l10n/po/gen/rank.inc
+++ buster/webwml/english/international/l10n/po/gen/rank.inc

--- stretch/webwml/english/international/l10n/po/gen/stats
+++ buster/webwml/english/international/l10n/po/gen/stats

--- stretch/webwml/english/international/l10n/po/rank.en.html
+++ buster/webwml/english/international/l10n/po/rank.en.html

There might be some reproducibility issues here too.


7. Changes related to image width/height attributes
===


--- stretch/webwml/english/partners/2018/index.en.html
+++ buster/webwml/english/partners/2018/index.en.html

-  
+  

--- stretch/webwml/english/partners/2019/index.en.html
+++ buster/webwml/english/partners/2019/index.en.html

-  
+  

-  
+  

--- stretch/webwml/english/partners/index.en.html
+++ buster/webwml/english/partners/index.en.html

-  
+  

-  
+  

For some reason, width/height were empty, and they seem to be stripped
now? Progress, maybe, except that might be hiding the root issue?


8. More ordering changes (architectures, DSAs)
==

--- stretch/webwml/english/releases/potato/installmanual.en.html
+++ buster/webwml/english/releases/potato/installmanual.en.html

--- stretch/webwml/english/releases/potato/releasenotes.en.html
+++ buster/webwml/english/releases/potato/releasenotes.en.html

--- stretch/webwml/english/releases/woody/installmanual.en.html
+++ buster/webwml/english/releases/woody/installmanual.en.html

--- stretch/webwml/english/releases/woody/releasenotes.en.html
+++ buster/webwml/english/releases/woody/releasenotes.en.html

--- stretch/webwml/english/security/crossreferences.en.html
+++ buster/webwml/english/security/crossreferences.en.html

-- stretch/webwml/english/security/ref-table.inc
++ buster