Re: Error with make doc
Folks, Carl wrote to me privately: > OK, so now I have a failure: > > I did > > ``` > rm -rf build > mkdir build > cd build > ../autogen.sh --currdir > ../configure &> configure.log > make -j10 CPU_COUNT=10 all > make doc LANGS="en" VERBOSE=1 CPU_COUNT=10 -j10 &> make.doc.log > ``` > And the make doc failed. [This is based on the branch 'dev/wl/build-fixes'.] The log file `snippet-names-c5b165525229969dd268a5b19f9f64fd.log` contains ``` GNU LilyPond 2.25.14 (running Guile 3.0) Forking into jobs: (39647 39646 39645 39644 39643 39642 39641 39640 39639 39638) job 9 terminated with signal: 6 job 8 terminated with signal: 6 job 7 terminated with signal: 6 job 6 terminated with signal: 6 job 5 terminated with signal: 6 job 4 terminated with signal: 6 job 3 terminated with signal: 6 job 2 terminated with signal: 6 job 1 terminated with signal: 6 job 0 terminated with signal: 6 fatal error: Children (9 8 7 6 5 4 3 2 1 0) exited with errors. ``` AFAIK, signal 6 is 'SIGABRT'. Note that the files `lilypond-multi-run-*.log` all contain thread_suspend failed which seems to be a hint that we have a problem with thread support on macOS (Carl runs on macOS Sonoma which identifies itself as Darwin 23.2 on arm64e). Any idea how to tackle this? BTW, `make doc` without `CPU_COUNT=10` succeeds on his computer. Werner
Re: Error with make doc
> I can confirm that I get the proper HTML pages, compiling with -j10 > on both make bytecode and make doc. Great! > It was delightful to see the proper lilypond website on my local > file system! Thank you so much for fixing this! Glad that I could help. I will now polish the MR as requested by Jonas. >> The last step – I hope – will be to find out why `CPU_COUNT` fails... >> > > I will see if I can make CPU_COUNT fail. If not, I guess the build > system is good on MacOS! The actual question is whether it works at all, i.e., whether setting `CPU_COUNT=10` makes your system really compile multiple `.ly` files in parallel. Given that I've seen thread_suspend failed in your log files I suspect that LilyPOnd simply switches to serial processing instead. If you don't get a timing difference (which means that my assumption is correct) I will prepare something more specific for further tests. Werner
Re: Error with make doc
On Mon, Feb 12, 2024 at 9:22 AM Werner LEMBERG wrote: > > v2.25.13-8-gbe52228a70 > > Please test again! Theoretically, you should now get proper HTML > pages. > I can confirm that I get the proper HTML pages, compiling with -j10 on both make bytecode and make doc. It was delightful to see the proper lilypond website on my local file system! Thank you so much for fixing this! > > The last step – I hope – will be to find out why `CPU_COUNT` fails... > I will see if I can make CPU_COUNT fail. If not, I guess the build system is good on MacOS! Thanks so much for your work on this! Carl
Re: Error with make doc
On Sat, Feb 10, 2024 at 7:20 AM Werner LEMBERG wrote: > > >> systems like macOS that neither have or use '/bin/bash'. > > > > Apple changed the default shell for new users to zsh about 5 years > > ago, but bash is still there. I don't oppose making the build > > scripts more portable, but let's not misunderstand the situation. > > OK, thanks – let's see whether the new code makes a build succeed on > Carl's computer; otherwise I have to continue my investigations. > > Note that on OpenBSD, `bash` resides in `/usr/local/bin` if you > install it via `pkg_add`. > > On my system, default bash is at /opt/local/bin/bash But I do have /bin/bash Carl
Re: Error with make doc
>> systems like macOS that neither have or use '/bin/bash'. > > Apple changed the default shell for new users to zsh about 5 years > ago, but bash is still there. I don't oppose making the build > scripts more portable, but let's not misunderstand the situation. OK, thanks – let's see whether the new code makes a build succeed on Carl's computer; otherwise I have to continue my investigations. Note that on OpenBSD, `bash` resides in `/usr/local/bin` if you install it via `pkg_add`. Werner
Re: Error with make doc
On 2024-02-10 07:52, Werner LEMBERG wrote: systems like macOS that neither have or use '/bin/bash'. Apple changed the default shell for new users to zsh about 5 years ago, but bash is still there. I don't oppose making the build scripts more portable, but let's not misunderstand the situation. -- Dan
Re: Error with make doc
>> If I click on the Notation link from that page, I get to: >> >> out-www/offline-root/Documentation/web/notation.html >> >> which has no css, but has lots of links. > > This shouldn't happen, and it means that your documentation build is > still not correct. Continuing with the analysis of Carl's build tarball it seems that the script `build-doc-tree.sh` doesn't work as expected. I've now converted it to a normal shell script (i.e., it no longer uses syntax specific to bash), which should hopefully increase its portability to systems like macOS that neither have or use '/bin/bash'. Carl, please do another run of the following commands based on the updated 'dev/wl/build-fixes' branch, then create and upload a tarball of your `build` directory as before and send me a link to it. ``` ./configure make bytecode -j10 LANGS="en" make doc LANGS="en" ``` Regarding the failing `make doc -j10`: > > "/Users/carl/Development/lilypond/build/out/lybook-db/ > > snippet-names-c5b165525229969dd268a5b19f9f64fd.ly"' returned > > non-zero exit status 1. > > Please try to find out which of the snippet files mentioned in the > above file failed. `*.log` files should be in directory > `out/lybook-db`, too, and the error message should be near the end > of one of the log files. > > I would need both the contents of the failing snippet file and the > corresponding error message. I still need this, BTW. Werner
Re: Error with make doc
> If I click on the Notation link from that page, I get to: > > out-www/offline-root/Documentation/web/notation.html > > which has no css, but has lots of links. This shouldn't happen, and it means that your documentation build is still not correct. > I'm happy to send a tar file of my offline-root directory and > subdirectories if that would be helpful. Yes, something like that would help. However, I would like to see your *complete* (successful) build of the `bytecode` and `doc` make targets, with `LANGS="en"` to reduce space. If you don't mind, please issue the following command one level above your build directory: gtar cJvf build.tar.xz build where `gtar` is GNU tar (from the 'gnutar' MacPorts package), also assuming that your build directory is called `build`. Upload the created file `build.tar.xz` to a filesharing hoster of your choice and send me the link privately so that I can download the file for further inspection. For me, this archive has about 206MByte. [The `J` command makes GNU tar use the `xz` program for compression, which is much more efficient than the default `gz` (at the cost of compression speed), and which you probably have to install from MacPorts, too.] Werner
Re: Error with make doc
> "/Users/carl/Development/lilypond/build/out/lybook-db/ > snippet-names-c5b165525229969dd268a5b19f9f64fd.ly"' returned > non-zero exit status 1. Please try to find out which of the snippet files mentioned in the above file failed. `*.log` files should be in directory `out/lybook-db`, too, and the error message should be near the end of one of the log files. I would need both the contents of the failing snippet file and the corresponding error message. Werner
Re: Error with make doc
On Sun, Feb 4, 2024 at 7:21 AM Werner LEMBERG wrote: > > > Please test > > https://gitlab.com/lilypond/lilypond/-/merge_requests/2254 > > It should fix all the build problems you've reported. > This merge request allowed me to build the documentation using MacOS cp and find. I still got errors if I tried make doc with -j12. But without -j12 it succeeded. There are some minor issues with either the way the documentation is built or the way I am using it. IIRC, in the past, I could use Documentation/out-www/index.html as a beginning place for documentation, and I could get to all the manuals. Now, when I follow this link, I get into a documentation tree that has links for all the manuals, but no css nor images. It forwards me to out-www/offline-root/Documentation/web/index.html If I click on the Notation link from that page, I get to: out-www/offline-root/Documentation/web/notation.html which has no css, but has lots of links. Notation (split html) takes me to out-www/offline-root/Documentation/notation/index.html which also has no css, nor body text, nor images. On the notation.html page, if I click on the Notation (big html) page, I get to out-www/offline-root/Documentation/notation-big-page.html Which has appropriate css. If instead of the above, I go to the en tree version of Notation, I get Documentation/out-www/en/notation/index.html which has css, but no images. When I click on the "Back to documentation index" from that page, I get to Documentation/out-www/Documentation/web/manuals.html WHich does not exist. So it seems the one-big-page html docs have css and images, but not the single pages. I'm happy to send a tar file of my offline-root directory and subdirectories if that would be helpful. Thanks, Carl
Re: Error with make doc
On Sat, Feb 3, 2024 at 12:01 AM Werner LEMBERG wrote: > > [Carl sent me the file listing off-list, thanks! I will analyze it in > due course.] > > > [...] and make test with a single job (to avoid race conditions that > > have given me trouble in the past): > > Interesting. When is the 'past'? This shouldn't happen today, and it > would be great if you could try a `-jXX' make option in the future so > that we can see if you really get a failure. > It was current. And it remains current with the latest merge request. > > > now on to your requested steps: > > Ah, my steps were not intended to be used additionally but instead :-) > Ah, so I tried that. ``` ../configure make bytecode -j12 ``` Ran without error. ``` carl@carls-mbp-2 build % make doc -j12 CPU_COUNT=12 LANGS="en" Making input/regression/out-www/collated-files.list < 1888 files Making input/regression/out-www/collated-files.tely Making input/regression/out-www/collated-files.texi < tely Command '/Users/carl/Development/lilypond/build/out/bin/lilypond -I /Users/carl/Development/lilypond/input/regression -dseparate-page-formats=ps,pdf -djob-count=12 -dinclude-eps-fonts -dgs-load-fonts --header=doctitle --header=doctitleen --header=texidoc --header=texidocen -dcheck-internal-types -danti-alias-factor=2 -dfont-ps-resdir=/Users/carl/Development/lilypond/build/out-fonts -O TeX-GS -I "/Users/carl/Development/lilypond/input/regression" -I "/Users/carl/Development/lilypond/build/input/regression" -daux-files -dread-file-list -dno-strip-output-dir -dtall-page-formats=eps,png "/Users/carl/Development/lilypond/build/out/lybook-db/ snippet-names-c5b165525229969dd268a5b19f9f64fd.ly"' returned non-zero exit status 1. make[3]: *** [out-www/collated-files.texi] Error 1 make[2]: *** [WWW] Error 2 make[1]: *** [WWW] Error 2 make: *** [doc] Error 2 ``` I'm currently in the midst of a ``` make doc LANGS="en" ``` run. I think it will complete without error, but it's slow... If you tell me what I can run to provide information about the j12 error on make doc, I'm happy to do that and send you whatever you ask for. I'll send another email shortly with the results of the make doc run with only a single job. Thanks, Carl
Re: Error with make doc
>> > I omitted the '-l' option in the cp calls in the patch. >> > >> > 'make doc' succeeded without any errors, so I thought I had success. But >> > it turns out that >> > >> > a) I appear not to have the .css files copied (at least the output >> >showed no css -- just basic white screen), and >> > b) I appear not to have the image files copied (there are no images >> >that show up in any of the pages accessible from >> >www-out/offline-root). As it turned out, this is another incompatibility: `cp -a` doesn't work on macOS. Please test https://gitlab.com/lilypond/lilypond/-/merge_requests/2254 It should fix all the build problems you've reported. Werner
Re: Error with make doc
[Carl sent me the file listing off-list, thanks! I will analyze it in due course.] > [...] and make test with a single job (to avoid race conditions that > have given me trouble in the past): Interesting. When is the 'past'? This shouldn't happen today, and it would be great if you could try a `-jXX' make option in the future so that we can see if you really get a failure. > now on to your requested steps: Ah, my steps were not intended to be used additionally but instead :-) > Everything completed without error. This is good. Regarding the `cp -l` issue: Irrespective of the faulty implementation on macOS it turns out that option `-l` isn't portable at all, as I was told by the autoconf people. I'm going to replace it with something else that also works across file system boundaries (which even a working `cp -l` does not). Werner
Re: Error with make doc
On Fri, Feb 2, 2024 at 4:34 AM Werner LEMBERG wrote: > >> Please read this and check whether you are affected by this > >> problem. In case you can confirm the issue please try whether it > >> works if you omit the `-l` option in all `cp` calls in my patch. > > > > I checked -- the results of the problem were different for me than > > for the bug report. I got the error message, but the hard link was > > never copied (there were not two files with the same inode -- only > > the original file. > > OK, so there is indeed a problem with `cp -l` on your macOS. It's > still hard to believe. > Yes, and apparently it's a new bug (older versions of the OS didn't have it, IIUC). > > BTW, which macOS version are you using? > Sonoma 14.2.1 (23C71) > > > I omitted the '-l' option in the cp calls in the patch. > > > > 'make doc' succeeded without any errors, so I thought I had success. But > > it turns out that > > > > a) I appear not to have the .css files copied (at least the output > >showed no css -- just basic white screen), and > > b) I appear not to have the image files copied (there are no images > >that show up in any of the pages accessible from > >www-out/offline-root). > > It seems that my patch doesn't work as expected, or that I have missed > something. As usual, please start from scratch and run the standard > incantation (or your adapted variation of it) > > > ``` > ./configure > make bytecode -j12 > make doc -j12 CPU_COUNT=12 LANGS="en" > ``` > > (please tell me what git commit you are using so that I can try > exactly the same setup). After finishing, please call > > ``` > find . | gzip > find.log.gz > ``` > > in your build directory and send me the file *privately* for further > inspection (it's about 750kByte compressed – listing approx. 16 > files) and of no real interest to other readers of this list). > > I will do this today. > > Does it make sense to try again with the xargs part of the patch, > > but using the GNU cp and find? > > For you personally, it certainly makes sense so that you get a > successful build. For the LilyPond project, however, I think it's > best to use native OS tools so that the number of additional > requisites stays as small as possible. > I tried with the xargs part of the patch, with the GNU cp and find. It executed sucessfully (no errors). But it also had no css on the resulting index.html. I viewed the source of index.html. It referenced the css from css/lilypond-website.css, but there was no 'css' directory in build/out-www/offline-root/. I tried copying the css directory from lilypond/Documentation to build/out-www/offline-root, but still didn't get a properly styled webpage. Thanks, Carl
Re: Error with make doc
>> Please read this and check whether you are affected by this >> problem. In case you can confirm the issue please try whether it >> works if you omit the `-l` option in all `cp` calls in my patch. > > I checked -- the results of the problem were different for me than > for the bug report. I got the error message, but the hard link was > never copied (there were not two files with the same inode -- only > the original file. OK, so there is indeed a problem with `cp -l` on your macOS. It's still hard to believe. BTW, which macOS version are you using? > I omitted the '-l' option in the cp calls in the patch. > > 'make doc' succeeded without any errors, so I thought I had success. But > it turns out that > > a) I appear not to have the .css files copied (at least the output >showed no css -- just basic white screen), and > b) I appear not to have the image files copied (there are no images >that show up in any of the pages accessible from >www-out/offline-root). It seems that my patch doesn't work as expected, or that I have missed something. As usual, please start from scratch and run the standard incantation (or your adapted variation of it) ``` ./configure make bytecode -j12 make doc -j12 CPU_COUNT=12 LANGS="en" ``` (please tell me what git commit you are using so that I can try exactly the same setup). After finishing, please call ``` find . | gzip > find.log.gz ``` in your build directory and send me the file *privately* for further inspection (it's about 750kByte compressed – listing approx. 16 files) and of no real interest to other readers of this list). > Does it make sense to try again with the xargs part of the patch, > but using the GNU cp and find? For you personally, it certainly makes sense so that you get a successful build. For the LilyPond project, however, I think it's best to use native OS tools so that the number of additional requisites stays as small as possible. Werner
Re: Error with make doc
On Thu, Feb 1, 2024 at 10:47 AM Werner LEMBERG wrote: > > > Doing a bit of an internet search it seems this is a bug in the `cp` > implementation on macOS (as unbelievable it may be, given that `cp` is > such a fundamental command that exists since decades): > > > https://apple.stackexchange.com/questions/464706/why-do-i-get-bad-file-descriptor-when-copying-using-hardlink-flag > > Please read this and check whether you are affected by this problem. > In case you can confirm the issue please try whether it works if you > omit the `-l` option in all `cp` calls in my patch. > I checked -- the results of the problem were different for me than for the bug report. I got the error message, but the hard link was never copied (there were not two files with the same inode -- only the original file. I omitted the '-l' option in the cp calls in the patch. 'make doc' succeeded without any errors, so I thought I had success. But it turns out that a) I appear not to have the .css files copied (at least the output showed no css -- just basic white screen), and b) I appear not to have the image files copied (there are no images that show up in any of the pages accessible from www-out/offline-root). Does it make sense to try again with the xargs part of the patch, but using the GNU cp and find? Carl
Re: Error with make doc
> 'make doc' failed. > > It appears that under some conditions, we end up with a path > including '//'. Multiple slashes in succession are completely harmless; they are simply equivalent to `/`. Doing a bit of an internet search it seems this is a bug in the `cp` implementation on macOS (as unbelievable it may be, given that `cp` is such a fundamental command that exists since decades): https://apple.stackexchange.com/questions/464706/why-do-i-get-bad-file-descriptor-when-copying-using-hardlink-flag Please read this and check whether you are affected by this problem. In case you can confirm the issue please try whether it works if you omit the `-l` option in all `cp` calls in my patch. Werner
Re: Error with make doc
On Wed, Jan 31, 2024 at 10:53 PM Werner LEMBERG wrote: > > > Thanks a lot, very helpful. Please apply the attached patch to the > top-level `GNUmakefile.in` file, then retry from scratch (including > the `configure` step); it should catch all of the three problems > you've encountered. > It seems to work, but not perfectly. 'make all' and 'make test' work. 'make doc' failed. It appears that under some conditions, we end up with a path including '//'. Log file attached. Thanks, Carl make.doc.log.gz Description: GNU Zip compressed data
Re: Error with make doc
> I've attached three compressed log files. [...] Thanks a lot, very helpful. Please apply the attached patch to the top-level `GNUmakefile.in` file, then retry from scratch (including the `configure` step); it should catch all of the three problems you've encountered. The `-maxdepth` and `-mindepth` options of `find` are not completely portable according to the 'autoconf' info pages; however, we are using them in `aclocal.m4` since ages and nobody complained, so... Werner diff --git a/GNUmakefile.in b/GNUmakefile.in index 6c0e1f287d..b28130e779 100644 --- a/GNUmakefile.in +++ b/GNUmakefile.in @@ -153,7 +153,7 @@ $(outdir)/%-root/index.html: doc-build mkdir -p $(dir $@)/Documentation/ for d in . musicxml abc2ly lilypond-book ; do \ mkdir -p $(dir $@)/input/regression/$${d}; \ - cp --link -a input/regression/$${d}/out-www/* $(dir $@)/input/regression/$${d}/ ; \ + cp -l -a input/regression/$${d}/out-www/* $(dir $@)/input/regression/$${d}/ ; \ done find $(dir $@)/input/regression/ \ -type f \ @@ -164,18 +164,23 @@ $(outdir)/%-root/index.html: doc-build -not -name '*.mxl' \ -not -name 'collated-files.pdf' \ -delete - cp --link -a Documentation/out-www/webdoc/* $(dir $@)/Documentation/ - cp --link $(TOPDOC_TXT_FILES) $(dir $@) + cp -l -a Documentation/out-www/webdoc/* $(dir $@)/Documentation/ + cp -l $(TOPDOC_TXT_FILES) $(dir $@) # No link because the files come from the source directory. cp $(addprefix $(src-dir)/,$(README_FILES)) $(dir $@) $(PYTHON) $(buildscript-dir)/www_post.py $(PACKAGE_NAME) $(TOPLEVEL_VERSION) $(dir $@) "$*" find $(outdir)/$*-root -type l | xargs rm -f - cd $(outdir)/$*-root && find -name \*.html -execdir $(buildscript-dir)/fix-docsize.sh {} + - $(PERL) -i -pe 's#(href)="\.\./Documentation/#\1="#g' $(dir $@)/Documentation/*html - $(PERL) -i -pe 's#(href)="\.\./\.\./Documentation/#\1="../#g' $(dir $@)/Documentation/*/*html - # Fixup links to CSS files. - $(PERL) -i -pe 's#(href)="css/#\1="../../Documentation/css/#g' $(dir $@)/input/regression/*html - $(PERL) -i -pe 's#(href)="css/#\1="../../../Documentation/css/#g' $(dir $@)/input/regression/*/*html + cd $(outdir)/$*-root \ + && find . -name "*.html" -execdir $(buildscript-dir)/fix-docsize.sh {} + + find $(dir $@)/Documentation -maxdepth 1 -name "*html" \ + | xargs $(PERL) -i -pe 's#(href)="\.\./Documentation/#\1="#g' + find $(dir $@)/Documentation -mindepth 2 -maxdepth 2 -name "*html" \ + | xargs $(PERL) -i -pe 's#(href)="\.\./\.\./Documentation/#\1="../#g' + # Fix up links to CSS files. + find $(dir $@)/input/regression -maxdepth 1 -name "*html" \ + | xargs $(PERL) -i -pe 's#(href)="css/#\1="../../Documentation/css/#g' + find $(dir $@)/input/regression -mindepth 2 -maxdepth 2 -name "*html" \ + | xargs $(PERL) -i -pe 's#(href)="css/#\1="../../../Documentation/css/#g' doc-build: top-doc $(MAKE) out=www -C Documentation/ doc
Re: Error with make doc
On Wed, Jan 31, 2024 at 12:49 AM Werner LEMBERG wrote: > > > I already had to install (via homebrew) gnu coreutils and gnu > > findutils, since the MacOS 'cp' and 'find' commands don't work the > > same as the respective GNU commands. > > Please give more details. What problems did you experience with `cp` > and `find`? Did this happen during a call to `make ...` for LilyPond? > All errors happened during 'make doc'. They appeared to happen after all of the lilypond processing was done, while creating the html documentation pages. I've attached three compressed log files. One shows the cp error (when I was using MacOS cp command by default). The second shows the find error (after configuring my system to use GNU cp). The third shows the perl error (after configuring my system to use GNU find as well as GNU cp). This link shows an example of what might cause the cp error (as well as the homebrew way to avoid it): https://github.com/sahib/rmlint/issues/275 This link shows how to get the GNU find utilities on MacOS, but doesn't reference the particular error in the log file. It does, however, indicate that there are a number of differences between the MacOS find and the GNU find.: https://unix.stackexchange.com/questions/277914/find-not-working This link shows how the -name argument can lead to the observed error message: https://unix.stackexchange.com/questions/369771/find-command-not-working-illegal-option-n > This should work out of the box, which essentially means that we have > to either add tests to `configure` to catch the problematic tools, or > to rewrite the commands in the makefile or scripts to avoid issues > with GNU-style usage of those tools, for example. > > > I then get to a problem with perl: [...] > > Interesting. I wasn't aware that we ever call `perl` with a long list > of arguments during `make doc`. > > Please repeat this with saying > > ``` > make doc VERBOSE=1 &> make.doc.log > ``` > > so that we can more information in the log file. Usually, such errors > can easily be circumvented by using `xargs`. > That's the recommendation I found on the web, but I don't know the Makefile well enough to propose a patch using xargs. Thanks for all your work on this! Carl cp_error.make.doc.log.gz Description: GNU Zip compressed data find_error.make.doc.log.gz Description: GNU Zip compressed data perl_error.make.doc.log.gz Description: GNU Zip compressed data
Re: Error with make doc
> I already had to install (via homebrew) gnu coreutils and gnu > findutils, since the MacOS 'cp' and 'find' commands don't work the > same as the respective GNU commands. Please give more details. What problems did you experience with `cp` and `find`? Did this happen during a call to `make ...` for LilyPond? This should work out of the box, which essentially means that we have to either add tests to `configure` to catch the problematic tools, or to rewrite the commands in the makefile or scripts to avoid issues with GNU-style usage of those tools, for example. > I then get to a problem with perl: [...] Interesting. I wasn't aware that we ever call `perl` with a long list of arguments during `make doc`. Please repeat this with saying ``` make doc VERBOSE=1 &> make.doc.log ``` so that we can more information in the log file. Usually, such errors can easily be circumvented by using `xargs`. Werner
Re: Error with make doc
Still trying to get 'make doc' to complete successfully. At this point, I think I'm through all the lilypond stuff, and now I'm stuck in shell script errors. I already had to install (via homebrew) gnu coreutils and gnu findutils, since the MacOS 'cp' and 'find' commands don't work the same as the respective GNU commands. I then get to a problem with perl: carl@carls-mbp-2 build % make doc Making out-www/offline-root/index.html Processing HTML pages for offline target... /bin/sh: /opt/local/bin/perl: Argument list too long make[1]: *** [out-www/offline-root/index.html] Error 126 make[1]: *** Deleting file `out-www/offline-root/index.html' make: *** [doc] Error 2 The internet suggests I should look at getconf ARG_MAX: build % getconf ARG_MAX 1048576 This seems like it should not be a problem. Any suggestions for debugging this? Thanks, Carl
Re: Error with make doc
>> Aah, this looks like a bug in `typography-demo.ly`. Please try the >> following line in `typography-demo.ly` instead of the three >> original font lines. >> >> ``` >> property-defaults.fonts.serif = "Linux Libertine O, Noto Serif CJK JP, Noto >> Serif JP, serif" >> ``` > > I can confirm that this change allows 'make test' to complete > successfully. Thanks for testing. https://gitlab.com/lilypond/lilypond/-/merge_requests/2244 Werner
Re: Error with make doc
On Wed, Jan 24, 2024 at 11:49 PM Werner LEMBERG wrote: > > >> I installed Noto Serif JP, and added it to the fontcofig cache. > >> That got me through the previous file error. But then I got > >> stopped on another Japanese font (I suppose this might be a Linux > >> Libertine font, but my fc-list command shows that I have them > >> installed): > > Interesting. This shouldn't happen. The idea is that the set of > fonts used for building the documentation is closed and well-defined. > 'Noto Serif JP' is intended to cover all CJK testing stuff in > LilyPond. > > > I was able to successfully complete 'make test' by commenting out the > > following lines in input/regression/typography-demo.ly: > > > > % Font settings for Cyrillic and Hebrew > > % Linux Libertine fonts contain Cyrillic and Hebrew glyphs. > > %property-defaults.fonts.serif = "Linux Libertine O,serif" > > %property-defaults.fonts.sans = "Linux Biolinum O,sans-serif" > > %property-defaults.fonts.typewriter = "Linux Libertine Mono > O,monospace" > > Aah, this looks like a bug in `typography-demo.ly`. Please try the > following line in `typography-demo.ly` instead of the three original > font lines. > > ``` > property-defaults.fonts.serif = "Linux Libertine O, Noto Serif CJK JP, > Noto Serif JP, serif" > ``` > > I can confirm that this change allows 'make test' to complete successfully. Thanks! Carl
Re: Error with make doc
>> I installed Noto Serif JP, and added it to the fontcofig cache. >> That got me through the previous file error. But then I got >> stopped on another Japanese font (I suppose this might be a Linux >> Libertine font, but my fc-list command shows that I have them >> installed): Interesting. This shouldn't happen. The idea is that the set of fonts used for building the documentation is closed and well-defined. 'Noto Serif JP' is intended to cover all CJK testing stuff in LilyPond. > I was able to successfully complete 'make test' by commenting out the > following lines in input/regression/typography-demo.ly: > > % Font settings for Cyrillic and Hebrew > % Linux Libertine fonts contain Cyrillic and Hebrew glyphs. > %property-defaults.fonts.serif = "Linux Libertine O,serif" > %property-defaults.fonts.sans = "Linux Biolinum O,sans-serif" > %property-defaults.fonts.typewriter = "Linux Libertine Mono O,monospace" Aah, this looks like a bug in `typography-demo.ly`. Please try the following line in `typography-demo.ly` instead of the three original font lines. ``` property-defaults.fonts.serif = "Linux Libertine O, Noto Serif CJK JP, Noto Serif JP, serif" ``` Werner
Re: Error with make doc
On Tue, Jan 23, 2024 at 4:54 AM Carl Sorensen wrote: > > > On Mon, Jan 22, 2024 at 8:37 PM Carl Sorensen > wrote: > >> >> >> On Mon, Jan 22, 2024 at 5:58 PM Jean Abou Samra >> wrote: >> >>> Le lundi 22 janvier 2024 à 16:38 -0700, Carl Sorensen a écrit : >>> > > Looks like no error: >>> > > >>> > > carl@Carls-MBP-2 lilypond % build/out/bin/lilypond input/regression/ >>> pdf-copy-paste.ly >>> > > GNU LilyPond 2.25.13 (running Guile 3.0) >>> > > Processing `input/regression/pdf-copy-paste.ly' >>> > > Parsing... >>> > > Finding the ideal number of pages... >>> > > Fitting music on 1 page... >>> > > Drawing systems... >>> > > Converting to `pdf-copy-paste.pdf'... >>> > > Success: compilation successfully completed >>> >>> >>> Hmm. What if you run >>> >>> LC_ALL=C build/out/bin/lilypond input/regression/pdf-copy-paste.ly >>> >>> ? IIRC, tests are compiled with LC_ALL=C. >>> >> >> OK, I'll run this test. But it looks to me (see below).like I should be >> able to solve the problem by getting the Noto Serif JP font. I'll look at >> installing that font. >> > > I installed Noto Serif JP, and added it to the fontcofig cache. That got > me through the previous file error. But then I got stopped on another > Japanese font (I suppose this might be a Linux Libertine font, but my > fc-list command shows that I have them installed): > > carl@carls-mbp-2 build % cat out/lybook-testdb/1d/lily-2548dc6c.log > > Processing `1d/lily-2548dc6c.ly' > > Parsing... > > Renaming input to: `/Users/carl/Development/lilypond/input/regression/ > typography-demo.ly' > > Interpreting music...[8] > > Preprocessing graphical objects... > > Interpreting music... > > MIDI output to `1d/lily-2548dc6c.midi'... > > Finding the ideal number of pages... > > Fitting music on 1 page... > > Drawing systems... > > Writing header field `texidoc' to `1d/lily-2548dc6c.texidoc'... > > Layout output to `1d/lily-2548dc6c.eps'...ERROR: In procedure open-file: > In procedure open-file: No such file or directory: > "/System/Library/Fonts/??? ProN.ttc" > > I was able to successfully complete 'make test' by commenting out the following lines in input/regression/typography-demo.ly: % Font settings for Cyrillic and Hebrew % Linux Libertine fonts contain Cyrillic and Hebrew glyphs. %property-defaults.fonts.serif = "Linux Libertine O,serif" %property-defaults.fonts.sans = "Linux Biolinum O,sans-serif" %property-defaults.fonts.typewriter = "Linux Libertine Mono O,monospace" iI appears that if Lilypond needs to search for a font to display a unicode character, and that font has Japanese characters in the font name, then the font file is unable to be opened (as shown above). There's been some discussion between Jean and Werner indicating that it may be due to a locale problem. I suspect that an issue needs to be created, but I'm not sure how to do it. And the issue may only show up on MacOS, because people running Linux don't seem to be having this problem. I'd welcome suggestions about how to create an issue for this, but I've got a workaround that allows me to succeed with 'make test'. Thanks, Car
Re: Error with make doc
> carl@Carls-MBP-2 lilypond % tidy --version > > HTML Tidy for Mac OS X released on 31 October 2006 - Apple Inc. build 6141 This explains the failure, thanks: Your `tidy` version doesn't support HTML 5. I've submitted an MR to catch this. https://gitlab.com/lilypond/lilypond/-/merge_requests/2241 > I didn't see tidy listed as a prerequisite. It is not a prerequisite; it is simply used if available, as an additional aid to the developers. As it turns out, the naive test for checking the presence of the binary is not sufficient; we must also check for a proper version. Please be reminded that up to now *nobody* worked on building the (complete) documentation on the Mac! You are effectively entering new territory (for which I'm very thankful), and there are dragons, as you've already found out. Werner
Re: Error with make doc
On Mon, Jan 22, 2024 at 8:37 PM Carl Sorensen wrote: > > > On Mon, Jan 22, 2024 at 5:58 PM Jean Abou Samra > wrote: > >> Le lundi 22 janvier 2024 à 16:38 -0700, Carl Sorensen a écrit : >> > > Looks like no error: >> > > >> > > carl@Carls-MBP-2 lilypond % build/out/bin/lilypond input/regression/ >> pdf-copy-paste.ly >> > > GNU LilyPond 2.25.13 (running Guile 3.0) >> > > Processing `input/regression/pdf-copy-paste.ly' >> > > Parsing... >> > > Finding the ideal number of pages... >> > > Fitting music on 1 page... >> > > Drawing systems... >> > > Converting to `pdf-copy-paste.pdf'... >> > > Success: compilation successfully completed >> >> >> Hmm. What if you run >> >> LC_ALL=C build/out/bin/lilypond input/regression/pdf-copy-paste.ly >> >> ? IIRC, tests are compiled with LC_ALL=C. >> > > OK, I'll run this test. But it looks to me (see below).like I should be > able to solve the problem by getting the Noto Serif JP font. I'll look at > installing that font. > I installed Noto Serif JP, and added it to the fontcofig cache. That got me through the previous file error. But then I got stopped on another Japanese font (I suppose this might be a Linux Libertine font, but my fc-list command shows that I have them installed): carl@carls-mbp-2 build % cat out/lybook-testdb/1d/lily-2548dc6c.log Processing `1d/lily-2548dc6c.ly' Parsing... Renaming input to: `/Users/carl/Development/lilypond/input/regression/ typography-demo.ly' Interpreting music...[8] Preprocessing graphical objects... Interpreting music... MIDI output to `1d/lily-2548dc6c.midi'... Finding the ideal number of pages... Fitting music on 1 page... Drawing systems... Writing header field `texidoc' to `1d/lily-2548dc6c.texidoc'... Layout output to `1d/lily-2548dc6c.eps'...ERROR: In procedure open-file: In procedure open-file: No such file or directory: "/System/Library/Fonts/??? ProN.ttc" WHen I manually run lilypond on this file it succeeds: carl@carls-mbp-2 build % out/bin/lilypond out/lybook-testdb/1d/ lily-2548dc6c.ly GNU LilyPond 2.25.13 (running Guile 3.0) Processing `out/lybook-testdb/1d/lily-2548dc6c.ly' Parsing... Renaming input to: `/Users/carl/Development/lilypond/input/regression/ typography-demo.ly' Interpreting music...[8] Preprocessing graphical objects... Interpreting music... MIDI output to `lily-2548dc6c.midi'... Finding the ideal number of pages... Fitting music on 1 page... Drawing systems... Converting to `lily-2548dc6c.pdf'... Success: compilation successfully completed It seems the difference may be I'm not creating the .eps file when I run out/bin/lilypond. What command could I run to try to duplicate the error in the log file? Thanks, Carl
Re: Error with make doc
>> Can someone remind me why the build system sets the C locale in the >> first place? > > Answering my own question: I guess it is because we can't assume the > system has an en_US.UTF-8 locale or a C.UTF-8 locale? Exactly. Some time ago (see https://lists.gnu.org/archive/html/lilypond-devel/2022-11/msg00247.html and the following mails in the thread) I worked on fixing exactly that (see attached old patch), but I stopped since we could then – at least partially – circumvent the related problems. I also stopped working on a `local-c-utf8.m4` file (also attached) to recognize the 'C.UTF-8' and/or 'en_US.UTF-8' locales, which turned out to be a really hard problem. > Maybe we ought to set only LC_MESSAGES to C? It should prevent > overriding the system locale for file names but still make logs > output in English. Honestly, I don't know. The whole locale stuff is such a mess. Werner From 231f5c2fba0fccf333fc7873da771fc8493b71e2 Mon Sep 17 00:00:00 2001 From: Werner Lemberg Date: Mon, 7 Nov 2022 09:38:17 +0100 Subject: [PATCH 1/2] generic-vars.make: Fix locale for build process MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * In non-English indices of the PDF documentation, this fixes the initials for entries starting with a non-ASCII character. This is a generic problem with the `texindex` awk script (used by the Texinfo infrastructure to sort indices). https://lists.gnu.org/archive/html/bug-texinfo/2022-11/msg8.html * Some non-ASCII characters are now correctly displayed in info, PDF, and HTML output files (example: `rotated 45?` → `rotated 45°` in the documentation of the `\rotate` markup). Also update documentation accordingly. --- Documentation/en/contributor/lsr-work.itexi | 15 - Documentation/en/usage/running.itely| 2 +- Documentation/es/usage/running.itely| 3 ++- Documentation/fr/usage/running.itely| 2 +- Documentation/it/usage/running.itely| 2 +- Documentation/ja/usage/running.itely| 2 +- make/generic-vars.make | 25 ++--- python/book_latex.py| 2 +- python/book_texinfo.py | 2 +- 9 files changed, 39 insertions(+), 16 deletions(-) diff --git a/Documentation/en/contributor/lsr-work.itexi b/Documentation/en/contributor/lsr-work.itexi index b03ebf8122..c69d8ca485 100644 --- a/Documentation/en/contributor/lsr-work.itexi +++ b/Documentation/en/contributor/lsr-work.itexi @@ -445,12 +445,15 @@ updating the binary running the LSR. @item Download the latest snippet tarball from -@uref{https://lsr.di.unimi.it/download/} and extract it. -The relevant files can be found in the @file{all} subdirectory. -Make sure your shell is using an English language version, for -example @code{LANG=en_US}, then run @command{convert-ly} on all -the files. Use the command-line option @code{--to=version} to -ensure the snippets are updated to the correct stable version. +@uref{https://lsr.di.unimi.it/download/} and extract it; the +relevant files can be found in the @file{all} subdirectory. + +Now run @code{LANG=C.UTF-8 convert-ly} on all the files. The +@code{LANG=C.UTF-8} part sets the @code{LANG} environment variable +to a locale with UTF-8 encoding and disabled translations while +@command{convert-ly} gets executed. Also use the command-line +option @code{--to=@var{version}} to ensure the snippets are +updated to the correct stable version. @item Make sure that you are using @command{convert-ly} from the latest diff --git a/Documentation/en/usage/running.itely b/Documentation/en/usage/running.itely index a24c7ac392..029d1ae446 100644 --- a/Documentation/en/usage/running.itely +++ b/Documentation/en/usage/running.itely @@ -925,7 +925,7 @@ overrides the value derived from the location of the @item LANG The language for LilyPond data sent to @code{stdout} and @code{stderr}, for example progress reports, warning messages, or -debug output. Example: @code{LANG=de}. +debug output. Example: @samp{LANG=de_DE.UTF-8}. @item LILYPOND_LOGLEVEL The default loglevel. If LilyPond is called without an explicit diff --git a/Documentation/es/usage/running.itely b/Documentation/es/usage/running.itely index 7f2629cd7d..7c722c50af 100644 --- a/Documentation/es/usage/running.itely +++ b/Documentation/es/usage/running.itely @@ -977,7 +977,8 @@ ubicación del binario @command{lilypond}. @item LANG Idioma de los datos de LilyPond enviados a @code{stdout} y a @code{stderr}, por ejemplo informes del avance, mensajes de -advertencia o salida de depuración. Ejemplo: @code{LANG=es}. +advertencia o salida de depuración. Ejemplo: +@samp{LANG=es_ES.UTF-8}. @item LILYPOND_LOGLEVEL Nivel de registro predeterminado. Si LilyPond se llama sin ningún diff --git a/Documentation/fr/usage/running.itely b/Documentation/fr/usage/running.itely index 5295ab0638..752ace6a2a 100644 ---
Re: Error with make doc
On Mon, Jan 22, 2024 at 5:58 PM Jean Abou Samra wrote: > Le lundi 22 janvier 2024 à 16:38 -0700, Carl Sorensen a écrit : > > > Looks like no error: > > > > > > carl@Carls-MBP-2 lilypond % build/out/bin/lilypond input/regression/ > pdf-copy-paste.ly > > > GNU LilyPond 2.25.13 (running Guile 3.0) > > > Processing `input/regression/pdf-copy-paste.ly' > > > Parsing... > > > Finding the ideal number of pages... > > > Fitting music on 1 page... > > > Drawing systems... > > > Converting to `pdf-copy-paste.pdf'... > > > Success: compilation successfully completed > > > Hmm. What if you run > > LC_ALL=C build/out/bin/lilypond input/regression/pdf-copy-paste.ly > > ? IIRC, tests are compiled with LC_ALL=C. > OK, I'll run this test. But it looks to me (see below).like I should be able to solve the problem by getting the Noto Serif JP font. I'll look at installing that font. Here's the results of the test: carl@carls-mbp-2 lilypond % LC_ALL=C build/out/bin/lilypond input/regression/pdf-copy-paste.ly GNU LilyPond 2.25.13 (running Guile 3.0) Processing `input/regression/pdf-copy-paste.ly' Parsing... Finding the ideal number of pages... Fitting music on 1 page... Drawing systems... Converting to `pdf-copy-paste.pdf'... Success: compilation successfully completed carl@carls-mbp-2 lilypond % As you can see, no error. I'll work on the Noto JP Sans. THanks, Carl
Re: Error with make doc
> Can someone remind me why the build system sets the C locale > in the first place? Answering my own question: I guess it is because we can't assume the system has an en_US.UTF-8 locale or a C.UTF-8 locale? Maybe we ought to set only LC_MESSAGES to C? It should prevent overriding the system locale for file names but still make logs output in English. signature.asc Description: This is a digitally signed message part
Re: Error with make doc
Ah, but, wait. $ touch àéù.txt $ guile3.0 scheme@(guile-user)> (open-file "àéù.txt" "r") $1 = # ~/tmp $ LC_ALL=C guile3.0 scheme@(guile-user)> (open-file "àéù.txt" "r") ice-9/boot-9.scm:1685:16: In procedure raise-exception: In procedure open-file: No such file or directory: "??.txt" OK, so Guile is assuming that the locale encoding determines the encoding of file names (which is quite reasonable), and so, when run under the C locale, it's transforming non-ASCII characters into question marks before resolving the path, as part of converting to the locale encoding. Can someone remind me why the build system sets the C locale in the first place? signature.asc Description: This is a digitally signed message part
Re: Error with make doc
Le lundi 22 janvier 2024 à 18:00 -0700, Carl Sorensen a écrit : > So I ran the file in this log message: > > > carl@carls-mbp-2 build % out/bin/lilypond out/share/lilypond/current/ly/ > init.ly > > GNU LilyPond 2.25.13 (running Guile 3.0) > > Processing `out/share/lilypond/current/ly/init.ly' > > Parsing... > > out/share/lilypond/current/ly/init.ly:47:1: error: \maininput not allowed > outside init files > > > \maininput > > out/share/lilypond/current/ly/init.ly:1: warning: no \version statement > found, please add > > > \version "2.25.13" > > > for future compatibility > > out/share/lilypond/current/ly/init.ly:1: warning: no \version statement > found, please add > > > \version "2.25.13" > > > for future compatibility > > fatal error: failed files: "out/share/lilypond/current/ly/init.ly" > > > Now I have found an error in the file -- it appears ist is due to > \maininput not being allowed. This isn't the problem. init.ly is a special file that LilyPond always loads before starting to compile any file and is not meant to be compiled standalone like this. The reason the error message mentions init.ly is that init.ly contains some Scheme code that is used to trigger the compilation. (This could be improved.) The problem really is the font issue in the error message. signature.asc Description: This is a digitally signed message part
Re: Error with make doc
Le mardi 23 janvier 2024 à 01:58 +0100, Jean Abou Samra a écrit : > The latter is basically that this regtest (pdf-copy-paste.ly) > has Japanese text, specifying the Noto Serif JP font, but since > you apparently don't have that font installed, Fontconfig falls > back to some system font, and apparently there is an encoding > issue somewhere with the file name of that font but I don't know > where exactly. Like Werner, I did some tests by renaming fonts > on my machine, and there were no issues, even with LC_ALL=C. (NB: The "??" in the error message is not indicative of a problem per se. If the environment configures LC_ALL=C then Guile detects that the current locale is not Unicode-aware and falls back to printing non-ASCII characters as "?". This doesn't necessarily mean that the string object being printed actually has these question marks, it could be perfectly correct.) signature.asc Description: This is a digitally signed message part
Re: Error with make doc
On Mon, Jan 22, 2024 at 4:37 PM Carl Sorensen wrote: > > > On Sun, Jan 21, 2024 at 12:33 AM Werner LEMBERG wrote: > >> >> >> You have to check the log files in `lybook-db` to find the problematic >> >> `.ly` file(s). Does `make test` succeed? You should actually try that >> >> before running `make doc`... >> >> >> > >> > Interesting -- the CG has make doc show up before make test. But I >> > appreciate the suggestion, so I decided to restart >> > >> > So I did a make clean, and then a make test. >> >> I've never seen the errors you are reporting – as Dan writes, it may >> be an issue with the `tidy` program. However, I'm using version 5.4.0 >> on my GNU/Linux box while MacPorts offers the more recent version >> 5.8.0; maybe there is indeed a problem in the created HTML code that >> wasn't detected or handled by tidy in earlier versions. >> >> Another possibility is that macOS comes with an ancient `tidy` version >> that doesn't understand HTML 5 stuff like ``, and you don't >> have installed `tidy` via MacPorts at all. In this case we have to >> update the `output-distance` script to check the `tidy` version before >> actually using it... >> > > That looks like an issue.Apparently my 'tidy' is not installed by > MacPorts. > > carl@carls-mbp-2 ly % which tidy > /usr/bin/tidy > > and > > carl@Carls-MBP-2 lilypond % tidy --version > > HTML Tidy for Mac OS X released on 31 October 2006 - Apple Inc. build 6141 > > > > I didn't see tidy listed as a prerequisite. > > After installing tidy in MacPorts, I get > > carl@carls-mbp-2 ~ % which tidy > > /opt/local/bin/tidy > > > and > > > carl@carls-mbp-2 ~ % tidy --version > > HTML Tidy for Apple macOS version 5.8.0 > > > I'll try rerunning 'make test' and see where I end up. I think that you > have clearly helped me identify a problem with tidy. > > OK. I deleted the build directory, and redid autogen.sh and ../configure. And I also did make dist-clean (because I was asked to do so). ../configure succeeded. 'make all' succeeded. 'make test' gave this error: Processing /Users/carl/Development/lilypond/build/out/lybook-testdb/ snippet-names-ece0c512da6daf7d9aff45809ce97556.ly Command '/Users/carl/Development/lilypond/build/out/bin/lilypond -I /Users/carl/Development/lilypond/input/regression -dseparate-page-formats=ps -dseparate-log-files -ddeterministic -dinclude-eps-fonts -dgs-load-fonts --header=texidoc -dcheck-internal-types -danti-alias-factor=1 -I "/Users/carl/Development/lilypond/input/regression" -I "/Users/carl/Development/lilypond/build/input/regression" -daux-files -dread-file-list -dno-strip-output-dir -dtall-page-formats=eps "/Users/carl/Development/lilypond/build/out/lybook-testdb/ snippet-names-ece0c512da6daf7d9aff45809ce97556.ly"' returned non-zero exit status 1. make[2]: *** [out-test/collated-files.texi] Error 1 make[1]: *** [lysdoc-test] Error 2 make: *** [test] Error 2 I tested the command that returned a non-zero exit status, and it succeeded. carl@carls-mbp-2 build % /Users/carl/Development/lilypond/build/out/bin/lilypond-I /Users/carl/Development/lilypond/input/regression -dseparate-page-formats=ps -dseparate-log-files -ddeterministic -dinclude-eps-fonts -dgs-load-fonts --header=texidoc -dcheck-internal-types -danti-alias-factor=1 -I "/Users/carl/Development/lilypond/input/regression" -I "/Users/carl/Development/lilypond/build/input/regression" -daux-files -dread-file-list -dno-strip-output-dir -dtall-page-formats=eps "/Users/carl/Development/lilypond/build/out/lybook-testdb/ snippet-names-cce821d5aaa438c29a8c2ee16cd514b0.ly" GNU LilyPond 2.25.13 (running Guile 3.0) Processing `0f/lily-acefd898.ly' I did the grep command suggested in the 'make test' output: % grep -L systems.texi out/lybook-testdb/*/*log | sed s/log/ly/g | xargs grep -H sourcefilename out/lybook-testdb/0f/lily-acefd898.ly:\sourcefilename "/Users/carl/Development/lilypond/input/regression/pdf-copy-paste.ly" I ran the file identified in the grep output: carl@carls-mbp-2 build % out/bin/lilypond ../input/regression/ pdf-copy-paste.ly GNU LilyPond 2.25.13 (running Guile 3.0) Processing `/Users/carl/Development/lilypond/build/out/share/lilypond/current/ly/../input/regression/ pdf-copy-paste.ly' Parsing... Finding the ideal number of pages... Fitting music on 1 page... Drawing systems... Converting to `pdf-copy-paste.pdf'... Success: compilation successfully completed I did the 'find' command suggested in the 'make test' output: carl@carls-mbp-2 build % find out/lybook-testdb -name "*.log" | xargs grep -H "\(error\|warning\)" out/lybook-testdb/0f/lily-acefd898.log:/Users/carl/Development/lilypond/build/out/share/lilypond/current/ly/init.ly:66:2: error: Guile signaled an error for the expression beginning here So I ran the file in this log message: carl@carls-mbp-2 build % out/bin/lilypond out/share/lilypond/current/ly/ init.ly GNU LilyPond 2.25.13 (running Guile 3.0) Processing `out/share/lilypond/current/ly/init.ly'
Re: Error with make doc
Le lundi 22 janvier 2024 à 16:38 -0700, Carl Sorensen a écrit : > > Looks like no error: > > > > carl@Carls-MBP-2 lilypond % build/out/bin/lilypond > > input/regression/pdf-copy-paste.ly > > GNU LilyPond 2.25.13 (running Guile 3.0) > > Processing `input/regression/pdf-copy-paste.ly' > > Parsing... > > Finding the ideal number of pages... > > Fitting music on 1 page... > > Drawing systems... > > Converting to `pdf-copy-paste.pdf'... > > Success: compilation successfully completed Hmm. What if you run LC_ALL=C build/out/bin/lilypond input/regression/pdf-copy-paste.ly ? IIRC, tests are compiled with LC_ALL=C. > > On Werner's earlier email, he indicated there may be a problem with tidy. > > I've gone ahead and installed tidy from MacPorts, and will try redoing > > 'make test' again. You have a problem with tidy *and* a problem with your fonts. The latter is basically that this regtest (pdf-copy-paste.ly) has Japanese text, specifying the Noto Serif JP font, but since you apparently don't have that font installed, Fontconfig falls back to some system font, and apparently there is an encoding issue somewhere with the file name of that font but I don't know where exactly. Like Werner, I did some tests by renaming fonts on my machine, and there were no issues, even with LC_ALL=C. signature.asc Description: This is a digitally signed message part
Re: Error with make doc
On Mon, Jan 22, 2024 at 3:58 PM Jean Abou Samra wrote: > Le lundi 22 janvier 2024 à 21:10 +, Werner LEMBERG a écrit : > > > > > > OK, thanks. This means that there is apparently a macOS-specific > > problem with either Guile or LilyPond. For testing purposes I renamed > > one of my local fonts to something containing Katakana, and my > > self-compiled LilyPond binary could load this font without any glitch. > > > > Jean, Jonas, any idea how to proceed? > > > Carl, please check that > > build/out/bin/lilypond input/regression/pdf-copy-paste.ly Looks like no error: carl@Carls-MBP-2 lilypond % build/out/bin/lilypond input/regression/ pdf-copy-paste.ly GNU LilyPond 2.25.13 (running Guile 3.0) Processing `input/regression/pdf-copy-paste.ly' Parsing... Finding the ideal number of pages... Fitting music on 1 page... Drawing systems... Converting to `pdf-copy-paste.pdf'... Success: compilation successfully completed So it appears to me that there may be something with texidoc that is generating the error. On Werner's earlier email, he indicated there may be a problem with tidy. I've gone ahead and installed tidy from MacPorts, and will try redoing 'make test' again. > > > gives the same error, then run > > build/out/bin/lilypond -ddebug-eval input/regression/pdf-copy-paste.ly > > so we have a more precise error location. > > Also send the output of the `locale` command. > carl@carls-mbp-2 Fonts % locale LANG="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_CTYPE="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_ALL= Thanks! Carl
Re: Error with make doc
On Sun, Jan 21, 2024 at 12:33 AM Werner LEMBERG wrote: > > >> You have to check the log files in `lybook-db` to find the problematic > >> `.ly` file(s). Does `make test` succeed? You should actually try that > >> before running `make doc`... > >> > > > > Interesting -- the CG has make doc show up before make test. But I > > appreciate the suggestion, so I decided to restart > > > > So I did a make clean, and then a make test. > > I've never seen the errors you are reporting – as Dan writes, it may > be an issue with the `tidy` program. However, I'm using version 5.4.0 > on my GNU/Linux box while MacPorts offers the more recent version > 5.8.0; maybe there is indeed a problem in the created HTML code that > wasn't detected or handled by tidy in earlier versions. > > Another possibility is that macOS comes with an ancient `tidy` version > that doesn't understand HTML 5 stuff like ``, and you don't > have installed `tidy` via MacPorts at all. In this case we have to > update the `output-distance` script to check the `tidy` version before > actually using it... > That looks like an issue.Apparently my 'tidy' is not installed by MacPorts. carl@carls-mbp-2 ly % which tidy /usr/bin/tidy and carl@Carls-MBP-2 lilypond % tidy --version HTML Tidy for Mac OS X released on 31 October 2006 - Apple Inc. build 6141 I didn't see tidy listed as a prerequisite. After installing tidy in MacPorts, I get carl@carls-mbp-2 ~ % which tidy /opt/local/bin/tidy and carl@carls-mbp-2 ~ % tidy --version HTML Tidy for Apple macOS version 5.8.0 I'll try rerunning 'make test' and see where I end up. I think that you have clearly helped me identify a problem with tidy. Thanks, Carl
Re: Error with make doc
Le lundi 22 janvier 2024 à 21:10 +, Werner LEMBERG a écrit : > > > > OK, thanks. This means that there is apparently a macOS-specific > problem with either Guile or LilyPond. For testing purposes I renamed > one of my local fonts to something containing Katakana, and my > self-compiled LilyPond binary could load this font without any glitch. > > Jean, Jonas, any idea how to proceed? Carl, please check that build/out/bin/lilypond input/regression/pdf-copy-paste.ly gives the same error, then run build/out/bin/lilypond -ddebug-eval input/regression/pdf-copy-paste.ly so we have a more precise error location. Also send the output of the `locale` command. Thanks. signature.asc Description: This is a digitally signed message part
Re: Error with make doc
>> > In procedure open-file: No such file or directory: >> > "/System/Library/Fonts/??? W4.ttc" [...] >> > >> > SO it looks to me like there's a problem with not getting UTF characters >> > into the proper file name in the Scheme procedure. Any ideas for what to >> > do next? >> >> What does `fc-list` report? Do you see correct file names? >> (`fc-list` should come with the FontConfig library). > > Shows the fonts with the appropriate kana/kanji names: > > carl@carls-mbp-2 Fonts % fc-list | grep W4 > > /System/Library/Fonts/ヒラギノ丸ゴ ProN W4.ttc: Hiragino Maru Gothic ProN, > [...] OK, thanks. This means that there is apparently a macOS-specific problem with either Guile or LilyPond. For testing purposes I renamed one of my local fonts to something containing Katakana, and my self-compiled LilyPond binary could load this font without any glitch. Jean, Jonas, any idea how to proceed? Werner
Re: Error with make doc
On Sun, Jan 21, 2024 at 11:05 PM Werner LEMBERG wrote: > > In procedure open-file: No such file or directory: > > "/System/Library/Fonts/??? W4.ttc" > > > > I checked out the /System/Library/Fonts directory, and found these fonts > > listed: > > > > ヒラギノ角ゴシック W3.ttc > > > > ヒラギノ角ゴシック W4.ttc > > > > SO it looks to me like there's a problem with not getting UTF characters > > into the proper file name in the Scheme procedure. Any ideas for what to > > do next? > > What does `fc-list` report? Do you see correct file names? > (`fc-list` should come with the FontConfig library). > Shows the fonts with the appropriate kana/kanji names: carl@carls-mbp-2 Fonts % fc-list | grep W4 /System/Library/Fonts/ヒラギノ丸ゴ ProN W4.ttc: Hiragino Maru Gothic ProN,ヒラギノ丸ゴ ProN,Hiragino Maru Gothic ProN W4,ヒラギノ丸ゴ ProN W4:style=W4,Regular /System/Library/Fonts/ヒラギノ角ゴシック W4.ttc: Hiragino Sans,ヒラギノ角ゴシック,Hiragino Sans W4,ヒラギノ角ゴシック W4:style=W4,Regular /System/Library/Fonts/ヒラギノ角ゴシック W4.ttc: .Hiragino Kaku Gothic Interface:style=W4 /System/Library/Fonts/ヒラギノ丸ゴ ProN W4.ttc: Hiragino Maru Gothic Pro,ヒラギノ丸ゴ Pro,Hiragino Maru Gothic Pro W4,ヒラギノ丸ゴ Pro W4:style=W4,Regular Thanks, Carl
Re: Error with make doc
> In procedure open-file: No such file or directory: > "/System/Library/Fonts/??? W4.ttc" > > I checked out the /System/Library/Fonts directory, and found these fonts > listed: > > ヒラギノ角ゴシック W3.ttc > > ヒラギノ角ゴシック W4.ttc > > SO it looks to me like there's a problem with not getting UTF characters > into the proper file name in the Scheme procedure. Any ideas for what to > do next? What does `fc-list` report? Do you see correct file names? (`fc-list` should come with the FontConfig library). Werner
Re: Error with make doc
Thanks for the hint, Dan! I got to a new log file that gave me what I believe is a helpful error message: carl@carls-mbp-2 build % cat out/lybook-testdb/0f/lily-acefd898.log Processing `0f/lily-acefd898.ly' Parsing... Renaming input to: `/Users/carl/Development/lilypond/input/regression/ pdf-copy-paste.ly' Writing header field `texidoc' to `0f/lily-acefd898.texidoc'... Layout output to `0f/lily-acefd898.eps'... /Users/carl/Development/lilypond/build/out/share/lilypond/current/ly/init.ly:66:2: error: Guile signaled an error for the expression beginning here # (let ((book-handler (if (defined? 'default-toplevel-book-handler) In procedure open-file: No such file or directory: "/System/Library/Fonts/??? W4.ttc" I checked out the /System/Library/Fonts directory, and found these fonts listed: ヒラギノ角ゴシック W3.ttc ヒラギノ角ゴシック W4.ttc SO it looks to me like there's a problem with not getting UTF characters into the proper file name in the Scheme procedure. Any ideas for what to do next? Thanks, Carl > >
Re: Error with make doc
>> You have to check the log files in `lybook-db` to find the problematic >> `.ly` file(s). Does `make test` succeed? You should actually try that >> before running `make doc`... >> > > Interesting -- the CG has make doc show up before make test. But I > appreciate the suggestion, so I decided to restart > > So I did a make clean, and then a make test. I've never seen the errors you are reporting – as Dan writes, it may be an issue with the `tidy` program. However, I'm using version 5.4.0 on my GNU/Linux box while MacPorts offers the more recent version 5.8.0; maybe there is indeed a problem in the created HTML code that wasn't detected or handled by tidy in earlier versions. Another possibility is that macOS comes with an ancient `tidy` version that doesn't understand HTML 5 stuff like ``, and you don't have installed `tidy` via MacPorts at all. In this case we have to update the `output-distance` script to check the `tidy` version before actually using it... Werner
Re: Error with make doc
On 2024-01-20 20:55, Carl Sorensen wrote: File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 524, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/usr/bin/tidy', '-o', '/dev/null', '-q', 'compare-dir1dir2/index.html']' returned non-zero exit status 2. Try commenting out this part of scripts/build/output-distance.py and retrying: tidy_bin = shutil.which('tidy') if tidy_bin: subprocess.run([tidy_bin, '-o', '/dev/null', '-q', html_fn], check=True) I'm not sure that that's the only problem, but it should eliminate one thing. -- Dan
Re: Error with make doc
On Thu, Jan 18, 2024 at 11:05 PM Werner LEMBERG wrote: > > > "/Users/carl/Development/lilypond/build/out/lybook-db/ > > snippet-names-ece0c512da6daf7d9aff45809ce97556.ly"' returned non-zero > exit > > status 1. > > You have to check the log files in `lybook-db` to find the problematic > `.ly` file(s). Does `make test` succeed? You should actually try that > before running `make doc`... > Interesting -- the CG has make doc show up before make test. But I appreciate the suggestion, so I decided to restart So I did a make clean, and then a make test. Got an error, and this is the log file ( /my_path/lilypond/build/scripts/build/./out/output-distance-test.log ): Command: /Library/Frameworks/Python.framework/Versions/3.10/bin/python3 -W default -W ignore::ResourceWarning /Users/carl/Development/lilypond/scripts/build/output-distance.py --test --output-dir=./out GNU LilyPond 2.25.13 (running Guile 3.0) Processing `20' Processing `19' Processing `20grob' Processing `20multipage' Processing `19multipage' Processing `added' Processing `removed' Processing `added-multipage.ly' Success: compilation successfully completed line 96 column 7 - Error: is not recognized! line 96 column 7 - Warning: discarding unexpected line 97 column 1 - Error: is not recognized! line 97 column 1 - Warning: discarding unexpected line 99 column 1 - Warning: discarding unexpected line 100 column 1 - Error: is not recognized! line 100 column 1 - Warning: discarding unexpected line 100 column 57 - Warning: discarding unexpected line 101 column 1 - Warning: discarding unexpected line 109 column 7 - Error: is not recognized! line 109 column 7 - Warning: discarding unexpected line 110 column 1 - Error: is not recognized! line 110 column 1 - Warning: discarding unexpected line 112 column 1 - Warning: discarding unexpected line 114 column 1 - Error: is not recognized! test results in ./out/test-output-distance comparing dir1 to dir2 25 eps 2 midi 7 log 1 gittxt 35 total no source for dir2/context.ly no source for dir2/removed.ly no source for dir2/midi-differ.midi no source for dir2/log-differ.log no source for dir2/removed.log no source for dir2/tree.gittxt converting 4 EPS files converting 16 EPS files converting 1 EPS files output-distance summary: 13 changed 3 in baseline only 0 below threshold 3 unchanged Traceback (most recent call last): File "/my_path/lilypond/scripts/build/output-distance.py", line 1424, in main() File "/my_path/lilypond/scripts/build/output-distance.py", line 1412, in main run_tests() File "/my_path/lilypond/scripts/build/output-distance.py", line 1359, in run_tests test_compare_tree_pairs() File "/my_path/lilypond/scripts/build/output-distance.py", line 1199, in test_compare_tree_pairs subprocess.run([tidy_bin, '-o', '/dev/null', '-q', html_fn], check=True) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 524, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/usr/bin/tidy', '-o', '/dev/null', '-q', 'compare-dir1dir2/index.html']' returned non-zero exit status 2. I'm not sure how to interpret or track down this error further. Any help would be appreciated. Thanks, Carl
Re: Error with make doc
> "/Users/carl/Development/lilypond/build/out/lybook-db/ > snippet-names-ece0c512da6daf7d9aff45809ce97556.ly"' returned non-zero exit > status 1. You have to check the log files in `lybook-db` to find the problematic `.ly` file(s). Does `make test` succeed? You should actually try that before running `make doc`... BTW, if you execute `make test`, you are greeted with the following advice: ``` To begin investigating regression-test crashes, try this: grep -L systems.texi out/lybook-testdb/*/*log | sed s/log/ly/g | xargs grep -H sourcefilename Matching lines may provide original test source file names. The .ly files in which they appear are preprocessed test input. The corresponding .log files may have errors or warnings. If the above is not helpful, try sifting through this: find out/lybook-testdb -name "*.log" | xargs grep -H "\(error\|warning\)" ``` which can also be used for finding problems with `make doc` (but you have to replace `lybook-testdb` with `lybook-db` in the above commands). Werner
Re: Error from "make doc"
Federico, On 22/05/16 21:37, Federico Bruni wrote: Il giorno dom 22 mag 2016 alle 11:52, James Loweha scritto: Install the package lmodern: sudo apt-get install lmodern Added to LilyDev only one month ago: https://github.com/fedelibre/LilyDev/commit/2d4ed72d0d0a5f1b73b71d80489524c624e5727f I am confused here. Had John not used LilyDev but followed the documented CG instructions for whatever Linux distribution - had he had one already installed - then he wouldn't have hit this issue I suspect, I say that as someone who compiles doc most days on both Ubuntu and Fedora and have tested and documented all the instructions for all the distributions listed in the CG and on my own 'at home' Fedora install I don't see any 'modern' fonts installed if I run the command: fc-list | grep modern (or 'grep lm') I can also see that the XeteX package listed in Debian's repos shows lmodern as 'recommended' however on my own system I have texlive-xetex installed and yet I still don't see lmodern installed, so it must not be a dep on Fedora either. lmodern is not a req as far as I can tell. So what is special about LilyDev? I can only say that when I tried to build the doc with xetex installed (and lmdodern not installed) I got this error in LilyDev/Debian: For xetex backend Debian package lmodern is needed In Fedora the package is called texlive-lm. This command returns a list of the installed files: rpm -ql texlive-lm I can only guess that these fonts are supposed to be available only to Texlive and not to the whole system?! Yes that seems correct when I look at my system I do have those fonts. So it also seems that the texlive-xetex package and the various dependencies are different across one or more distributions - Linux eh? 'Doncha juss lovvit?' Going back through my checkins for the CG I also note that we didn't suggest XeTeX when I did all those different distributions, so we *do* need to make some additional caveats. I'll create a ticket. James ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Error from "make doc"
Il giorno dom 22 mag 2016 alle 11:52, James Loweha scritto: Install the package lmodern: sudo apt-get install lmodern Added to LilyDev only one month ago: https://github.com/fedelibre/LilyDev/commit/2d4ed72d0d0a5f1b73b71d80489524c624e5727f I am confused here. Had John not used LilyDev but followed the documented CG instructions for whatever Linux distribution - had he had one already installed - then he wouldn't have hit this issue I suspect, I say that as someone who compiles doc most days on both Ubuntu and Fedora and have tested and documented all the instructions for all the distributions listed in the CG and on my own 'at home' Fedora install I don't see any 'modern' fonts installed if I run the command: fc-list | grep modern (or 'grep lm') I can also see that the XeteX package listed in Debian's repos shows lmodern as 'recommended' however on my own system I have texlive-xetex installed and yet I still don't see lmodern installed, so it must not be a dep on Fedora either. lmodern is not a req as far as I can tell. So what is special about LilyDev? I can only say that when I tried to build the doc with xetex installed (and lmdodern not installed) I got this error in LilyDev/Debian: For xetex backend Debian package lmodern is needed In Fedora the package is called texlive-lm. This command returns a list of the installed files: rpm -ql texlive-lm I can only guess that these fonts are supposed to be available only to Texlive and not to the whole system?!? ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Error from "make doc"
Federico, On 22/05/16 08:43, Federico Bruni wrote: I don't think so. It's supposed to work out-of-the-box in LilyDev, but I forgot to include this package in LilyDev 4.1. See warning in the release note: https://github.com/fedelibre/LilyDev/releases/tag/4.1 Il giorno dom 22 mag 2016 alle 3:54, John Gourlayha scritto: Thank you! Installing the package lmodern fixed the problem. Shouldn’t this instruction be part of the Contributor’s Guide? On May 20, 2016, at 1:36 AM, Federico Bruni wrote: Il giorno ven 20 mag 2016 alle 5:10, John Gourlay ha scritto: I’m trying to set up a new LilyDev development system after some kind of VirtualBox error that trashed my old one. I’ve run “make” successfully in my ~/lilypond-git/build directory, and lilypond runs. When I run “make doc”, however, I get errors. One of the messages from make directed me to look at a log file named suffix-lyxml.dblatex.log. This file contains the following messages about invalid fonts. Have I overlooked something in setting up my system? Install the package lmodern: sudo apt-get install lmodern Added to LilyDev only one month ago: https://github.com/fedelibre/LilyDev/commit/2d4ed72d0d0a5f1b73b71d80489524c624e5727f I am confused here. Had John not used LilyDev but followed the documented CG instructions for whatever Linux distribution - had he had one already installed - then he wouldn't have hit this issue I suspect, I say that as someone who compiles doc most days on both Ubuntu and Fedora and have tested and documented all the instructions for all the distributions listed in the CG and on my own 'at home' Fedora install I don't see any 'modern' fonts installed if I run the command: fc-list | grep modern (or 'grep lm') I can also see that the XeteX package listed in Debian's repos shows lmodern as 'recommended' however on my own system I have texlive-xetex installed and yet I still don't see lmodern installed, so it must not be a dep on Fedora either. lmodern is not a req as far as I can tell. So what is special about LilyDev? -- -- James ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Error from "make doc"
I don't think so. It's supposed to work out-of-the-box in LilyDev, but I forgot to include this package in LilyDev 4.1. See warning in the release note: https://github.com/fedelibre/LilyDev/releases/tag/4.1 Il giorno dom 22 mag 2016 alle 3:54, John Gourlayha scritto: Thank you! Installing the package lmodern fixed the problem. Shouldn’t this instruction be part of the Contributor’s Guide? On May 20, 2016, at 1:36 AM, Federico Bruni wrote: Il giorno ven 20 mag 2016 alle 5:10, John Gourlay ha scritto: I’m trying to set up a new LilyDev development system after some kind of VirtualBox error that trashed my old one. I’ve run “make” successfully in my ~/lilypond-git/build directory, and lilypond runs. When I run “make doc”, however, I get errors. One of the messages from make directed me to look at a log file named suffix-lyxml.dblatex.log. This file contains the following messages about invalid fonts. Have I overlooked something in setting up my system? Install the package lmodern: sudo apt-get install lmodern Added to LilyDev only one month ago: https://github.com/fedelibre/LilyDev/commit/2d4ed72d0d0a5f1b73b71d80489524c624e5727f ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Error from "make doc"
Thank you! Installing the package lmodern fixed the problem. Shouldn’t this instruction be part of the Contributor’s Guide? > On May 20, 2016, at 1:36 AM, Federico Bruniwrote: > > Il giorno ven 20 mag 2016 alle 5:10, John Gourlay > ha scritto: >> I’m trying to set up a new LilyDev development system after some kind of >> VirtualBox error that trashed my old one. I’ve run “make” successfully in my >> ~/lilypond-git/build directory, and lilypond runs. When I run “make doc”, >> however, I get errors. One of the messages from make directed me to look at >> a log file named suffix-lyxml.dblatex.log. This file contains the following >> messages about invalid fonts. Have I overlooked something in setting up my >> system? > > Install the package lmodern: > > sudo apt-get install lmodern > > Added to LilyDev only one month ago: > https://github.com/fedelibre/LilyDev/commit/2d4ed72d0d0a5f1b73b71d80489524c624e5727f > ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Error from "make doc"
Il giorno ven 20 mag 2016 alle 5:10, John Gourlayha scritto: I’m trying to set up a new LilyDev development system after some kind of VirtualBox error that trashed my old one. I’ve run “make” successfully in my ~/lilypond-git/build directory, and lilypond runs. When I run “make doc”, however, I get errors. One of the messages from make directed me to look at a log file named suffix-lyxml.dblatex.log. This file contains the following messages about invalid fonts. Have I overlooked something in setting up my system? Install the package lmodern: sudo apt-get install lmodern Added to LilyDev only one month ago: https://github.com/fedelibre/LilyDev/commit/2d4ed72d0d0a5f1b73b71d80489524c624e5727f ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel