Re: Today's problem with GUB build
David Kastrup writes: > Jonas Hahnfeld writes: > >> Am Mittwoch, den 15.07.2020, 21:35 +0200 schrieb David Kastrup: >>> Jonas Hahnfeld writes: >>> > Am Mittwoch, den 15.07.2020, 17:31 +0100 schrieb Phil Holmes: >>> > > Here's the logfile and the ly file. >>> > >>> > Could this be collisions of the random file names generated for >>> > temporary files? The argument to backend-library.scm:248 comes >>> > from create-file-exclusive which returns #f if the file already exists >>> > (or could not be created). >>> >>> I had commented on the respective issue without response that the >>> parallel processes, without taking additional measures, will generate >>> the same "random" sequence, making this no better than just using >>> sequential numbers. >> >> "additional measures" are in place: multi-fork calls randomize-rand- >> seed *after* forking. The seed is initialized based on the current >> timestamp (might be the same) and the pid (different in the course of >> one run). We can still have collisions, but the amount of trouble (or >> rather the lack of reports until now) indicates that it is better than >> sequential numbers. This was discussed (and answered) in the review. > > Well ok. But only 100 random numbers are being used (there is > another call using 1000 instead, the choice appearing random). > Let's assume we have 10 processes going through 138 files each. The > processes are going to switch to the next output file asynchronously, so > with any change, there is a chance of the old number colliding with the > other processes' numbers, and the new number colliding. The probability > that a new number is different from an existing set of 9 is > 91/100. If we do this switch 1380 times, the probability of a > collision during one run is 1-(91/100)^1380, about 1 in 80. > > Now if I remember correctly, there were some changes in how > lilypond-book worked that typically resulted in double the number of > processes getting spawned than asked for which would give us 19 instead > of 9 possibilities for collision. That would raise the probability of a > collision to about 1 in 40 runs. Not using random at all but using the pid, in contrast, should be collision-proof, assuming that we are not working on a shared file system accessed by multiple computers with separate process id pools. But then locking is likely to be non-working anyway. -- David Kastrup
Re: Today's problem with GUB build
Jonas Hahnfeld writes: > Am Mittwoch, den 15.07.2020, 21:35 +0200 schrieb David Kastrup: >> Jonas Hahnfeld writes: >> > Am Mittwoch, den 15.07.2020, 17:31 +0100 schrieb Phil Holmes: >> > > Here's the logfile and the ly file. >> > >> > Could this be collisions of the random file names generated for >> > temporary files? The argument to backend-library.scm:248 comes >> > from create-file-exclusive which returns #f if the file already exists >> > (or could not be created). >> >> I had commented on the respective issue without response that the >> parallel processes, without taking additional measures, will generate >> the same "random" sequence, making this no better than just using >> sequential numbers. > > "additional measures" are in place: multi-fork calls randomize-rand- > seed *after* forking. The seed is initialized based on the current > timestamp (might be the same) and the pid (different in the course of > one run). We can still have collisions, but the amount of trouble (or > rather the lack of reports until now) indicates that it is better than > sequential numbers. This was discussed (and answered) in the review. Well ok. But only 100 random numbers are being used (there is another call using 1000 instead, the choice appearing random). Let's assume we have 10 processes going through 138 files each. The processes are going to switch to the next output file asynchronously, so with any change, there is a chance of the old number colliding with the other processes' numbers, and the new number colliding. The probability that a new number is different from an existing set of 9 is 91/100. If we do this switch 1380 times, the probability of a collision during one run is 1-(91/100)^1380, about 1 in 80. Now if I remember correctly, there were some changes in how lilypond-book worked that typically resulted in double the number of processes getting spawned than asked for which would give us 19 instead of 9 possibilities for collision. That would raise the probability of a collision to about 1 in 40 runs. -- David Kastrup
Re: Today's problem with GUB build
Am Mittwoch, den 15.07.2020, 21:35 +0200 schrieb David Kastrup: > Jonas Hahnfeld writes: > > Am Mittwoch, den 15.07.2020, 17:31 +0100 schrieb Phil Holmes: > > > Here's the logfile and the ly file. > > > > Could this be collisions of the random file names generated for > > temporary files? The argument to backend-library.scm:248 comes > > from create-file-exclusive which returns #f if the file already exists > > (or could not be created). > > I had commented on the respective issue without response that the > parallel processes, without taking additional measures, will generate > the same "random" sequence, making this no better than just using > sequential numbers. "additional measures" are in place: multi-fork calls randomize-rand- seed *after* forking. The seed is initialized based on the current timestamp (might be the same) and the pid (different in the course of one run). We can still have collisions, but the amount of trouble (or rather the lack of reports until now) indicates that it is better than sequential numbers. This was discussed (and answered) in the review. signature.asc Description: This is a digitally signed message part
Re: Today's problem with GUB build
Jonas Hahnfeld writes: > Am Mittwoch, den 15.07.2020, 17:31 +0100 schrieb Phil Holmes: >> Here's the logfile and the ly file. > > Could this be collisions of the random file names generated for > temporary files? The argument to backend-library.scm:248 comes > from create-file-exclusive which returns #f if the file already exists > (or could not be created). I had commented on the respective issue without response that the parallel processes, without taking additional measures, will generate the same "random" sequence, making this no better than just using sequential numbers. -- David Kastrup
Re: Python coding style
Hi, Le 02/07/2020 à 09:05, Han-Wen Nienhuys a écrit : You can also do a global cleanup. We have done this in the past, but it has the disadvantage that it makes history (eg. git-blame) harder to read. So, I did the global cleanup: https://gitlab.com/lilypond/lilypond/-/merge_requests/251 We can take it one bit at a time. In particular, if you can provide a recipe that would reformat diffs, I can integrate that in my pre-commit hook, so all new code is compliant. I don't have that in my toolbox, sorry. The issue is that tools like autopep8 and YAPF work on ASTs, so they can't cope with incomplete bits of code. I don't know a safe way to pick just the part of the generated diff that deals with what you change (if you see what I mean). On Wed, Jul 1, 2020 at 2:03 PM Jean Abou Samra wrote: to me it's a no-brainer to introduce automatic formatting. I don't really care what standard it enforces, as long as it is automated. I would consider it, but it isn't a no-brainer to me: every automatic formatting tool has its issues, and I prefer formatting my code myself to format the way I think most human-readable.Anyway, let's rather discuss this later when we come to adding a linter to CI. - Use `string` instead of `str` as an identifier − `str` being a built-in type already. This would be a good rule to follow if the Python team doesn't keep adding identifiers to the top level scope. For example, in other projects I had a lot of code become non-compliant overnight when Python decided that binary formatting of number is worthy of the toplevel identifier 'bin'. I agree that Python evolves rapidly, but I think that `str` is common enough to warrant not using as a variable since you could need it as its actual built-in value in your function. On the other hand, I didn't listen to pylint complaining that we redefined the builtin `credits`. Cheers, Jean
Re: Today's problem with GUB build
Le 15/07/2020 à 19:44, Jean Abou Samra a écrit :
Hi,
Le 15/07/2020 à 18:31, Phil Holmes a écrit :
Here's the logfile and the ly file.
/home/gub/NewGub/gub/target/linux-x86/root/usr/share/lilypond/current/scm/backend-library.scm:248:5:
In procedure close-port in expression (close-port port):
/home/gub/NewGub/gub/target/linux-x86/root/usr/share/lilypond/current/scm/backend-library.scm:248:5:
Wrong type argument in position 1 (expecting port): #f
I recall having seen a failure of the same kind yesterday in a make
test. After a make test-clean, it all went fine. Could it be that you
need to clean the tree somehow like a make doc-clean?
Okay, I was plain wrong. Just got this after a make test-clean && make test:
testing /home/jean/repos/lilypond/scripts/build/output-distance.py
Making ly/out/welcome.pdf < Welcome_to_LilyPond.ly
Making ly/out/welcome.png < Welcome_to_LilyPond.ly
Making ly/out/welcome.svg < Welcome_to_LilyPond.ly
Making ly/out/welcome_eps.pdf < Welcome_to_LilyPond.ly
Please check the logfile
/home/jean/repos/lilypond/build/ly/welcome.pdf.log
for errors. Last 20 lines:
?: 9* [#]
In /home/jean/repos/lilypond/build/out/share/lilypond/current/scm/lily.scm:
1114: 10* [ly:parse-file
"/home/jean/repos/lilypond/ly/Welcome_to_LilyPond.ly"]
In /home/jean/repos/lilypond/build/out/share/lilypond/current/ly/init.ly:
76: 11* (let* ((book-handler #)) (cond (# #) (# #)))
79: 12 (cond (# #) (# #))
In
/home/jean/repos/lilypond/build/out/share/lilypond/current/scm/lily-library.scm:
...
247: 13 [ly:book-process # #< Output_def> #< Output_def> "welcome"]
In unknown file:
?: 14* [output-framework "welcome" # (#7fc20d78af60>) ()]
In
/home/jean/repos/lilypond/build/out/share/lilypond/current/scm/framework-ps.scm:
738: 15* (let* (# # # # ...) (if # #) (initialize-font-embedding) ...)
738: 16* [make-tmpfile "welcome"]
In
/home/jean/repos/lilypond/build/out/share/lilypond/current/scm/backend-library.scm:
244: 17 (let* ((name #) (port #) (bport #)) (close-port port) bport)
248: 18* [close-port {#f}]
/home/jean/repos/lilypond/build/out/share/lilypond/current/scm/backend-library.scm:248:5:
In procedure close-port in expression (close-port port):
/home/jean/repos/lilypond/build/out/share/lilypond/current/scm/backend-library.scm:248:5:
Wrong type argument in position 1 (expecting port): #f
make[1]: *** [/home/jean/repos/lilypond/build/.././ly/GNUmakefile:20:
welcome.pdf] Error 1
make[1]: *** Waiting for unfinished jobs
make: *** [/home/jean/repos/lilypond/build/../GNUmakefile.in:292 : test]
Erreur 2
The actual fix was… err… git clean -xdf and building from scratch.
Hope that helps in some way.
Jean
Re: Today's problem with GUB build
Hi, Le 15/07/2020 à 18:31, Phil Holmes a écrit : Here's the logfile and the ly file. /home/gub/NewGub/gub/target/linux-x86/root/usr/share/lilypond/current/scm/backend-library.scm:248:5: In procedure close-port in expression (close-port port): /home/gub/NewGub/gub/target/linux-x86/root/usr/share/lilypond/current/scm/backend-library.scm:248:5: Wrong type argument in position 1 (expecting port): #f I recall having seen a failure of the same kind yesterday in a make test. After a make test-clean, it all went fine. Could it be that you need to clean the tree somehow like a make doc-clean? Cheers, Jean
Re: Today's problem with GUB build
Am Mittwoch, den 15.07.2020, 17:31 +0100 schrieb Phil Holmes: > Here's the logfile and the ly file. Could this be collisions of the random file names generated for temporary files? The argument to backend-library.scm:248 comes from create-file-exclusive which returns #f if the file already exists (or could not be created). Forking into jobs: (21124 21123 21122 21121 21120 21119 21118 21117) logfile lilypond-multi-run-3.log (exit 1): Processing `./c4/lily-b2be0729.ly' Parsing... Interpreting music... Preprocessing graphical objects... Calculating line breaks... Drawing systems... Writing ./c4/lily-b2be0729-1.signature Layout output to `./c4/lily-b2be0729.eps'... Converting to `./c4/lily-b2be0729.pdf'... /home/gub/NewGub/gub/target/linux-x86/root/usr/share/lilypond/current/scm/backend-library.scm:248:5: In procedure close-port in expression (close-port port): /home/gub/NewGub/gub/target/linux-x86/root/usr/share/lilypond/current/scm/backend-library.scm:248:5: Wrong type argument in position 1 (expecting port): #f logfile lilypond-multi-run-1.log (exit 1): Processing `./c9/lily-745608fe.ly' Parsing... Interpreting music... Preprocessing graphical objects... Calculating line breaks... Drawing systems... Writing ./c9/lily-745608fe-1.signature Layout output to `./c9/lily-745608fe.eps'... Converting to `./c9/lily-745608fe.pdf'... /home/gub/NewGub/gub/target/linux-x86/root/usr/share/lilypond/current/scm/backend-library.scm:108:5: In procedure rename-file in expression (rename-file pdf-name dest): /home/gub/NewGub/gub/target/linux-x86/root/usr/share/lilypond/current/scm/backend-library.scm:108:5: No such file or directory fatal error: Children (3 1) exited with errors. signature.asc Description: This is a digitally signed message part
Re: Today's problem with GUB build
Here's the logfile and the ly file. Once we understand the issue, I'll wait until you say "go" for 21.4. -- Phil Holmes - Original Message - From: "Jonas Hahnfeld" To: "Phil Holmes" ; "Devel" Sent: Wednesday, July 15, 2020 4:31 PM Subject: Re: Today's problem with GUB build snippet-map-bcc309d5e1e5a6301f36814c1d4d872c.ly 03/lily-fe271a83.ly 1a/lily-8bf6a717.ly 1b/lily-078e147a.ly 2d/lily-6aba4240.ly 2f/lily-dfa3e9b3.ly 32/lily-6a5101eb.ly 3a/lily-6be5f72e.ly 3e/lily-8cc6ae6f.ly 3e/lily-c910bcd3.ly 4a/lily-53e1472d.ly 4b/lily-1cae6603.ly 4d/lily-81647c29.ly 4d/lily-b68bee77.ly 4f/lily-528ba80b.ly 5d/lily-00a8e953.ly 68/lily-0719c095.ly 69/lily-0c146c35.ly 73/lily-501a5a38.ly 75/lily-c07b1519.ly 77/lily-9e016ba0.ly 90/lily-558e4780.ly 94/lily-dec79207.ly 98/lily-d0ac7612.ly b2/lily-1446d680.ly b2/lily-26dfe8f9.ly b3/lily-cac0e6d3.ly c4/lily-8d95e596.ly c4/lily-b2be0729.ly c9/lily-3a3639cf.ly c9/lily-745608fe.ly d2/lily-61c512af.ly e7/lily-46341530.ly e7/lily-9d77bd0c.ly snippet-names-bcc309d5e1e5a6301f36814c1d4d872c.log Description: Binary data
Re: Today's problem with GUB build
Am Mittwoch, den 15.07.2020, 16:26 +0100 schrieb Phil Holmes: > Here's the lilypond-doc.log zipped. Do you happen to still have out/lybook-db/snippet-names- bcc309d5e1e5a6301f36814c1d4d872c.log around? In any case, it's odd that you're seeing issues with the very same base commit as 2.21.3 > I was doing a new release to get the documentation on the website looking > correct again. As I said: This won't be the case with the current release/unstable because it doesn't have my fix merged. In my opinion, we need to merge https://gitlab.com/lilypond/lilypond/-/merge_requests/246 to master and then do another release from there. > We've had problems in the past if the build number is not > incremented. > > If I don't get problems I can easily do another release with the updates you > mentioned. If I get me act together, I plan to do a release every couple of > weeks anyway. > > -- > Phil Holmes > > > - Original Message - > From: "Jonas Hahnfeld" < > hah...@hahnjo.de > > > To: "Phil Holmes" < > em...@philholmes.net > >; "Devel" < > lilypond-devel@gnu.org > > > Sent: Wednesday, July 15, 2020 4:09 PM > Subject: Re: Today's problem with GUB build > > signature.asc Description: This is a digitally signed message part
Re: Today's problem with GUB build
Here's the lilypond-doc.log zipped. I was doing a new release to get the documentation on the website looking correct again. We've had problems in the past if the build number is not incremented. If I don't get problems I can easily do another release with the updates you mentioned. If I get me act together, I plan to do a release every couple of weeks anyway. -- Phil Holmes - Original Message - From: "Jonas Hahnfeld" To: "Phil Holmes" ; "Devel" Sent: Wednesday, July 15, 2020 4:09 PM Subject: Re: Today's problem with GUB build <>
Re: Today's problem with GUB build
Am Mittwoch, den 15.07.2020, 15:04 +0100 schrieb Phil Holmes: > I get this: > > Making Documentation/out-www/web.texi (copy) > Command '/home/gub/NewGub/gub/target/linux-x86/root/usr/bin/lilypond \ > -dbackend=eps --formats=ps,png,pdf -djob-count=8 -dinclude-eps-fonts > -dgs-load-fonts > --header=doctitle --header=doctitleca --header=doctitlecs > --header=doctitlede > --header=doctitlees --header=doctitlefr --header=doctitlehu > --header=doctitleit > --header=doctitleja --header=doctitlenl --header=doctitlept > --header=doctitlezh > --header=texidoc --header=texidocca --header=texidoccs --header=texidocde > --header=texidoces > --header=texidocfr --header=texidochu --header=texidocit --header=texidocja > --header=texidocnl --header=texidocpt --header=texidoczh > -dcheck-internal-types > -ddump-signatures -danti-alias-factor=2 > -dfont-ps-resdir=/home/gub/NewGub/gub/target/linux-x86/build/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/out-fonts > > -O TeX-GS -I "./" -I > "/home/gub/NewGub/gub/target/linux-x86/build/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/Documentation" > > -I > "/home/gub/NewGub/gub/target/linux-x86/src/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/Documentation" > > -I > "/home/gub/NewGub/gub/target/linux-x86/build/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/Documentation/out-www" > > -I > "/home/gub/NewGub/gub/target/linux-x86/build/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/Documentation/snippets/out" > > -I > "/home/gub/NewGub/gub/target/linux-x86/src/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/Documentation/included" > > -I > "/home/gub/NewGub/gub/target/linux-x86/src/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/Documentation/pictures" > > -I > "/home/gub/NewGub/gub/target/linux-x86/src/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/Documentation" > > -I > "/home/gub/NewGub/gub/target/linux-x86/src/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/input/regression" > > --formats=eps -deps-box-padding=3.00 -dread-file-list > -dno-strip-output-dir > /home/gub/NewGub/gub/target/linux-x86/build/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/out/lybook-db/snippet-names-bcc309d5e1e5a6301f36814c1d4d872c.ly' > > returned non-zero exit status 1. > make[2]: *** [out-www/notation.texi] Error 1 > make[2]: *** Deleting file `out-www/notation.texi' > make[2]: *** Waiting for unfinished jobs > make[1]: *** [WWW-1] Error 2 > make: *** [doc-stage-1] Error 2 Without more detailed logs, that will be hard to guess. However, I don't get the purpose of releasing 2.21.4 from the same base commit as 2.21.3? release/unstable doesn't have the merged translations nor the fix for the documentation build in GUB. Did I miss something? (The reduction of PDF sizes by Masamichi-san would also be good to have in the next release.) (Part of the confusion might be my Pull Request for GUB, see https://github.com/gperciva/gub/pull/75. This has nothing to do with the documentation build which is related to the version of Perl.) Jonas signature.asc Description: This is a digitally signed message part
Today's problem with GUB build
I get this: Making Documentation/out-www/web.texi (copy) Command '/home/gub/NewGub/gub/target/linux-x86/root/usr/bin/lilypond \ -dbackend=eps --formats=ps,png,pdf -djob-count=8 -dinclude-eps-fonts -dgs-load-fonts --header=doctitle --header=doctitleca --header=doctitlecs --header=doctitlede --header=doctitlees --header=doctitlefr --header=doctitlehu --header=doctitleit --header=doctitleja --header=doctitlenl --header=doctitlept --header=doctitlezh --header=texidoc --header=texidocca --header=texidoccs --header=texidocde --header=texidoces --header=texidocfr --header=texidochu --header=texidocit --header=texidocja --header=texidocnl --header=texidocpt --header=texidoczh -dcheck-internal-types -ddump-signatures -danti-alias-factor=2 -dfont-ps-resdir=/home/gub/NewGub/gub/target/linux-x86/build/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/out-fonts -O TeX-GS -I "./" -I "/home/gub/NewGub/gub/target/linux-x86/build/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/Documentation" -I "/home/gub/NewGub/gub/target/linux-x86/src/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/Documentation" -I "/home/gub/NewGub/gub/target/linux-x86/build/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/Documentation/out-www" -I "/home/gub/NewGub/gub/target/linux-x86/build/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/Documentation/snippets/out" -I "/home/gub/NewGub/gub/target/linux-x86/src/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/Documentation/included" -I "/home/gub/NewGub/gub/target/linux-x86/src/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/Documentation/pictures" -I "/home/gub/NewGub/gub/target/linux-x86/src/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/Documentation" -I "/home/gub/NewGub/gub/target/linux-x86/src/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/input/regression" --formats=eps -deps-box-padding=3.00 -dread-file-list -dno-strip-output-dir /home/gub/NewGub/gub/target/linux-x86/build/lilypond-git.sv.gnu.org--lilypond.git-release-unstable/out/lybook-db/snippet-names-bcc309d5e1e5a6301f36814c1d4d872c.ly' returned non-zero exit status 1. make[2]: *** [out-www/notation.texi] Error 1 make[2]: *** Deleting file `out-www/notation.texi' make[2]: *** Waiting for unfinished jobs make[1]: *** [WWW-1] Error 2 make: *** [doc-stage-1] Error 2 -- Phil Holmes
Re: New release
Am Dienstag, den 14.07.2020, 09:05 +0200 schrieb Jonas Hahnfeld: > Am Montag, den 13.07.2020, 22:38 +0200 schrieb Jonas Hahnfeld: > > Am Montag, den 13.07.2020, 21:07 +0200 schrieb Han-Wen Nienhuys: > > > On Mon, Jul 13, 2020 at 7:49 PM Phil Holmes wrote: > > > > I created a new release today. I know that (again) I got the news > > > > wrong and > > > > will need to fix that. However, it looks like the web docs are > > > > probably not > > > > correctly picking up a style sheet (see > > > > http://lilypond.org/doc/v2.21/Documentation/contributor/index_1#Introduction-to-contributing > > > > as a simple example) - they're certainly not presented correctly. Could > > > > anyone say what has gone wrong and what needs to be done to fix it? > > > > > > the 2.21 docs are missing a tocframe section in the manpages. Could > > > this be a fallout of the recent texi2html work? > > > > Ouch, that doesn't look good, but probably related to my work. (For all > > of the changes, I looked at diffs of the HTML docs before and after my > > changes and all looked clean, but that's only my environment...) > > > > From what I saw during a quick look, it could be our init file wasn't > > loaded at all - for example, all cross-links are prefixed with the > > linked book. I don't remember seeing this locally (does this happen for > > anyone?), so one possible difference is the Perl version. GUB has > > version 5.10 which is ... ancient. It could be that I'm using > > *something* (no Perl expert) that breaks loading the file in GUB. I'll > > investigate tomorrow. > > Pretty good guess: > $ ../gub/target/tools/root/usr/bin/perl -c > Documentation/lilypond-texi2html.init > Bareword found where operator expected at > Documentation/lilypond-texi2html.init line 827, near "s// class="toc_current">/r" > syntax error at Documentation/lilypond-texi2html.init line 827, near > "s///r" > Documentation/lilypond-texi2html.init had compilation errors. > > Okay, so let's do it the old way: > https://gitlab.com/lilypond/lilypond/-/merge_requests/246 > (not tested with GUB yet, will do later today) Took until today, but I can confirm it works with GUB on my system. (Note to future me: Don't include "-doc" in your branch name when testing with GUB. Just don't. GUB will strip the suffix and complain that it can't find the branch - big surprise...) signature.asc Description: This is a digitally signed message part
