Re: XML to .ly and Lilypond, again
Whoops. I had responded before reading the new digests, apologies; seems David has discovered the root of the issue, thank you very much! Cheers, Leszek. On 16 May 2017 at 09:47, Leszek Wrońskiwrote: > Hi Robert, > > thanks! I tried your file and I get similar messages: > > - > musicxml2ly: Reading MusicXML from Kyrie_copy.xml ... > musicxml2ly: Converting to LilyPond expressions... > () > musicxml2ly: Output to `Kyrie_copy.ly' > Traceback (most recent call last): > File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3269, in ? > main() > File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3263, in main > voices = convert(filename, options) > File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3175, in convert > score_information.print_ly(printer) > File "/home/leszek/lilypond/usr/share/lilypond/current/python/musicexp.py", > line 855, in print_ly > self.format_header_strings(k, v, printer) > File "/home/leszek/lilypond/usr/share/lilypond/current/python/musicexp.py", > line 848, in format_header_strings > printer.newline() > File "/home/leszek/lilypond/usr/share/lilypond/current/python/musicexp.py", > line 123, in newline > self._file.write (self._line + '\n') > File "/home/leszek/lilypond/usr/lib/python2.4/codecs.py", line 501, in write > return self.writer.write(data) > File "/home/leszek/lilypond/usr/lib/python2.4/codecs.py", line 178, in write > data, consumed = self.encode(object, self.errors) > UnicodeDecodeError: 'utf8' codec can't decode byte 0xa9 in position > 19: unexpected code byte > -- > > I'm using Ubuntu 16.04 64 bit Python 2.7.12, and Lilypond 2.19.48. > > So is it the case that musicxml2ly is trying to apply the ''utf8' > codec' to the whole file while parts of it are conded in utf16? > > Cheers, > > Leszek. > > On 15 May 2017 at 16:43, Robert Blackstone > wrote: >> Hi Leszek, >> >> Thanks for your reaction to my post. >> Interesting, yes, but of little help to you. >> >> We could do a little test. >> >> What happens if you use the xlm-file that can be downloaded from this link >> here? >> >> https://dl.dropboxusercontent.com/u/512573/FileChute/Kyrie_copy.zip >> >> Best regards, >> Robert >> >> >> On 15 May 2017, at 08:22 , Leszek Wroński wrote: >> >> Hi Robert, >> >> thanks! It's interesting that all is fine on a Mac. I just wanted to >> mention that I'm only using Finale, that is, Windows, to produce the >> XML. I then switch to Ubuntu 16.04 to use musicxml2ly because that's >> the system I generally use. So the problem is also present on Linux. >> >> This is actually a nice practical reason to use a Mac ;) >> >> Cheers, >> >> Leszek. >> >> On 14 May 2017 at 12:42, Robert Blackstone >> wrote: >> >> Hi Leszek, >> >> Out of curiosity I downloaded and processed the Kyrie.xml, from the link you >> attached to your post of 12-05-17. >> I was curious in particular because I mostly use the same procedure for >> engraving with LilyPond as you do, starting with Finale, except that I'm on >> a Mac (OSX 10.9.5 also known as Mavericks). >> So I imported your file in Frescobaldi, and then copied and pasted it in >> TeXShop, my favourite editor, which processed your file without any problem >> (both with LilyPond version 2.18.2 and 2.19.58.1). >> >> You can find the results here: >> >> https://dl.dropboxusercontent.com/u/512573/FileChute/LilyPond-test_Kyrie_from_xml-file.zip >> >> So apparently for a Mac there is nothing wrong, neither with your xml-file >> nor with recent musicxml2ly. >> It may be of litte help to you (as long as you stay on Windows ;-)) but it >> could perhaps help you in narrowing down the search for the source of >> trouble in your system. >> >> Best regards >> >> Robert Blackstone >> >> >> >> >> On 12 May 2017, at 13:42 , Leszek Wroński wrote: >> >> Dear Guys, >> >> first, thank you very much for the replies! >> >> Second, for a little bit of detective work. I have not updated my >> Finale in years, so if it produced correct XML files in the past, it >> should do the same now (I always write using a similar really basic >> setup, since actual editing is to be done in Lilypond). Anyway, I >> tracked down an XML file which converted correctly in 2.19.5. I wanted >> to check whether I would get the 'lots of null characters' problem >> currently. To my surprise, I didn't even get to that part. Musicxml2ly >> gave me the following: >> >> -- >> musicxml2ly: Reading MusicXML from Kyrie.xml ... >> musicxml2ly: Converting to LilyPond expressions... >> (.) >> musicxml2ly: Output to `Kyrie.ly' >> Traceback (most recent call last): >> File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3269, in ? >> main() >> File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3263, in main >> voices = convert(filename, options) >> File
Re: XML to .ly and Lilypond, again
Hi Robert, thanks! I tried your file and I get similar messages: - musicxml2ly: Reading MusicXML from Kyrie_copy.xml ... musicxml2ly: Converting to LilyPond expressions... () musicxml2ly: Output to `Kyrie_copy.ly' Traceback (most recent call last): File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3269, in ? main() File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3263, in main voices = convert(filename, options) File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3175, in convert score_information.print_ly(printer) File "/home/leszek/lilypond/usr/share/lilypond/current/python/musicexp.py", line 855, in print_ly self.format_header_strings(k, v, printer) File "/home/leszek/lilypond/usr/share/lilypond/current/python/musicexp.py", line 848, in format_header_strings printer.newline() File "/home/leszek/lilypond/usr/share/lilypond/current/python/musicexp.py", line 123, in newline self._file.write (self._line + '\n') File "/home/leszek/lilypond/usr/lib/python2.4/codecs.py", line 501, in write return self.writer.write(data) File "/home/leszek/lilypond/usr/lib/python2.4/codecs.py", line 178, in write data, consumed = self.encode(object, self.errors) UnicodeDecodeError: 'utf8' codec can't decode byte 0xa9 in position 19: unexpected code byte -- I'm using Ubuntu 16.04 64 bit Python 2.7.12, and Lilypond 2.19.48. So is it the case that musicxml2ly is trying to apply the ''utf8' codec' to the whole file while parts of it are conded in utf16? Cheers, Leszek. On 15 May 2017 at 16:43, Robert Blackstonewrote: > Hi Leszek, > > Thanks for your reaction to my post. > Interesting, yes, but of little help to you. > > We could do a little test. > > What happens if you use the xlm-file that can be downloaded from this link > here? > > https://dl.dropboxusercontent.com/u/512573/FileChute/Kyrie_copy.zip > > Best regards, > Robert > > > On 15 May 2017, at 08:22 , Leszek Wroński wrote: > > Hi Robert, > > thanks! It's interesting that all is fine on a Mac. I just wanted to > mention that I'm only using Finale, that is, Windows, to produce the > XML. I then switch to Ubuntu 16.04 to use musicxml2ly because that's > the system I generally use. So the problem is also present on Linux. > > This is actually a nice practical reason to use a Mac ;) > > Cheers, > > Leszek. > > On 14 May 2017 at 12:42, Robert Blackstone > wrote: > > Hi Leszek, > > Out of curiosity I downloaded and processed the Kyrie.xml, from the link you > attached to your post of 12-05-17. > I was curious in particular because I mostly use the same procedure for > engraving with LilyPond as you do, starting with Finale, except that I'm on > a Mac (OSX 10.9.5 also known as Mavericks). > So I imported your file in Frescobaldi, and then copied and pasted it in > TeXShop, my favourite editor, which processed your file without any problem > (both with LilyPond version 2.18.2 and 2.19.58.1). > > You can find the results here: > > https://dl.dropboxusercontent.com/u/512573/FileChute/LilyPond-test_Kyrie_from_xml-file.zip > > So apparently for a Mac there is nothing wrong, neither with your xml-file > nor with recent musicxml2ly. > It may be of litte help to you (as long as you stay on Windows ;-)) but it > could perhaps help you in narrowing down the search for the source of > trouble in your system. > > Best regards > > Robert Blackstone > > > > > On 12 May 2017, at 13:42 , Leszek Wroński wrote: > > Dear Guys, > > first, thank you very much for the replies! > > Second, for a little bit of detective work. I have not updated my > Finale in years, so if it produced correct XML files in the past, it > should do the same now (I always write using a similar really basic > setup, since actual editing is to be done in Lilypond). Anyway, I > tracked down an XML file which converted correctly in 2.19.5. I wanted > to check whether I would get the 'lots of null characters' problem > currently. To my surprise, I didn't even get to that part. Musicxml2ly > gave me the following: > > -- > musicxml2ly: Reading MusicXML from Kyrie.xml ... > musicxml2ly: Converting to LilyPond expressions... > (.) > musicxml2ly: Output to `Kyrie.ly' > Traceback (most recent call last): > File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3269, in ? > main() > File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3263, in main > voices = convert(filename, options) > File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3175, in convert > score_information.print_ly(printer) > File "/home/leszek/lilypond/usr/share/lilypond/current/python/musicexp.py", > line 855, in print_ly > self.format_header_strings(k, v, printer) > File "/home/leszek/lilypond/usr/share/lilypond/current/python/musicexp.py", > line 848, in format_header_strings > printer.newline() >
Re: XML to .ly and Lilypond, again
On Fri 12 May 2017 at 11:59:05 (+0100), Phil Holmes wrote: > - Original Message - From: "Urs Liska" <u...@openlilylib.org> > To: <lilypond-user@gnu.org>; <bug-lilyp...@gnu.org> > Sent: Friday, May 12, 2017 11:00 AM > Subject: Re: XML to .ly and Lilypond, again > >Am 12.05.2017 um 11:26 schrieb Jan-Peter Voigt: > >>Dear Leszek, > >> > >>when I look with hexdump, I see a lot of NUL-bytes inside the file. > >>And it seems to me that the strings are in UTF-16 and the rest in > >>latin-1 or the like. At least it seems like a mixture of encodings. > > > >I think this is exactly what was discussed in the mentioned recent thread. > >So it seems this is an issue in musicxml2ly that has recently been > >introduced. Does anyone know or can anyone find out when this happened > >(by trying musicxml2ly from different LilyPond versions)? > > > >Urs > > See https://sourceforge.net/p/testlilyissues/issues/4751/ Took me a while to look at this, as I'm not used to this site. The places to check are: https://codereview.appspot.com/295120043/diff/1/python/musicexp.py green lines 138 and 139. https://codereview.appspot.com/295120043/diff/1/python/utilities.py (green) lines 66 through 72. Cheers, David. ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
On Sun 14 May 2017 at 08:33:26 (+0200), zs.has...@gmx.de wrote: > >On Fri 12 May 2017 at 12:03:36 (+0200), David Kastrup wrote: He wrote something else. But I wrote… > >My guess is that the wrong encoding comes from "original" strings > >being passed from the input to the output as opposed to all the > >normal strings that are generated by the conversion program. These > >original strings would carry their encoding with them. > If you are meaning finale is producing the wrong xml, I can tell you > Sibelius, musescore and capella also have this problem. No, they all should have the problem as they all write correct Unicode XML files. The fault was in the python program, because an enhancement was made before the version was new enough to handle it corrctly. Cheers, David. ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again, with a fix
On Sun 14 May 2017 at 15:35:51 (+0100), Phil Holmes wrote: > - Original Message - From: "Urs Liska" <u...@openlilylib.org> > To: <lilypond-user@gnu.org> > Sent: Sunday, May 14, 2017 3:06 PM > Subject: Re: XML to .ly and Lilypond, again > >Am 14.05.2017 um 16:03 schrieb Phil Holmes: > >>I've just confirmed Ian Ring's suggestion - removing the copyright > >>symbol allows the conversion to continue, but results in text with > >>spurious null characters. Only some of the text, as I reported in http://lists.gnu.org/archive/html/lilypond-user/2017-05/msg00241.html > >But can that be? Shouldn't MusicXML allow arbitrary regular Unicode > >characters? > > My understanding is the XML is like HTML and requires special > characters to be escaped. No, the norm is for XML to be written in Unicode as this one is, hence its header: So the program should be handling all the data in unicode, and the problem is the exact opposite of what I started out looking for. Handling unicode is tricky at best in python2, and I avoided it myself by switching to python3 before trying to do anything more than printing unicode to output, which is all musicxml2py should really be doing. However, the new version tries to do one clever thing and it's in split_string_and_preserve_doublequoted_substrings in utilities.py. This uses the shlex module whose preamble runs: The shlex class makes it easy to write lexical analyzers for simple syntaxes resembling that of the Unix shell. This will often be useful for writing minilanguages, (for example, in run control files for Python applications) or for parsing quoted strings. Prior to Python 2.7.3, this module did not support Unicode input. So the fate of the copyright symbol in printer.dump should be to go from u'"\xa9"' ← a unicode value to [u'"\xa9"'] ← a list with one unicode value but instead it gets mangled to ['"\x00\xa9\x00"', '\x00'] ← a list of ascii strings. I don't know what the change was meant to fix as I've never used musicxml in anger. But the easiest patch to get things to work is to replace words = utilities.split_string_and_preserve_doublequoted_substrings (str) with words = string.split (str) in .../lilypond-2.19.…/lilypond/usr/share/lilypond/current/python/utilities.py assuming you're running a downloaded version rather than one included in your distribution. (Debian is still installing 2.18 IIRC.) Cheers, David. ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
Hi Leszek, Thanks for your reaction to my post. Interesting, yes, but of little help to you. We could do a little test. What happens if you use the xlm-file that can be downloaded from this link here? https://dl.dropboxusercontent.com/u/512573/FileChute/Kyrie_copy.zip Best regards, Robert On 15 May 2017, at 08:22 , Leszek Wrońskiwrote: > Hi Robert, > > thanks! It's interesting that all is fine on a Mac. I just wanted to > mention that I'm only using Finale, that is, Windows, to produce the > XML. I then switch to Ubuntu 16.04 to use musicxml2ly because that's > the system I generally use. So the problem is also present on Linux. > > This is actually a nice practical reason to use a Mac ;) > > Cheers, > > Leszek. > > On 14 May 2017 at 12:42, Robert Blackstone > wrote: >> Hi Leszek, >> >> Out of curiosity I downloaded and processed the Kyrie.xml, from the link you >> attached to your post of 12-05-17. >> I was curious in particular because I mostly use the same procedure for >> engraving with LilyPond as you do, starting with Finale, except that I'm on >> a Mac (OSX 10.9.5 also known as Mavericks). >> So I imported your file in Frescobaldi, and then copied and pasted it in >> TeXShop, my favourite editor, which processed your file without any problem >> (both with LilyPond version 2.18.2 and 2.19.58.1). >> >> You can find the results here: >> >> https://dl.dropboxusercontent.com/u/512573/FileChute/LilyPond-test_Kyrie_from_xml-file.zip >> >> So apparently for a Mac there is nothing wrong, neither with your xml-file >> nor with recent musicxml2ly. >> It may be of litte help to you (as long as you stay on Windows ;-)) but it >> could perhaps help you in narrowing down the search for the source of >> trouble in your system. >> >> Best regards >> >> Robert Blackstone >> >> >> >> >> On 12 May 2017, at 13:42 , Leszek Wroński wrote: >> >> Dear Guys, >> >> first, thank you very much for the replies! >> >> Second, for a little bit of detective work. I have not updated my >> Finale in years, so if it produced correct XML files in the past, it >> should do the same now (I always write using a similar really basic >> setup, since actual editing is to be done in Lilypond). Anyway, I >> tracked down an XML file which converted correctly in 2.19.5. I wanted >> to check whether I would get the 'lots of null characters' problem >> currently. To my surprise, I didn't even get to that part. Musicxml2ly >> gave me the following: >> >> -- >> musicxml2ly: Reading MusicXML from Kyrie.xml ... >> musicxml2ly: Converting to LilyPond expressions... >> (.) >> musicxml2ly: Output to `Kyrie.ly' >> Traceback (most recent call last): >> File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3269, in ? >> main() >> File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3263, in main >> voices = convert(filename, options) >> File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3175, in convert >> score_information.print_ly(printer) >> File "/home/leszek/lilypond/usr/share/lilypond/current/python/musicexp.py", >> line 855, in print_ly >> self.format_header_strings(k, v, printer) >> File "/home/leszek/lilypond/usr/share/lilypond/current/python/musicexp.py", >> line 848, in format_header_strings >> printer.newline() >> File "/home/leszek/lilypond/usr/share/lilypond/current/python/musicexp.py", >> line 123, in newline >> self._file.write (self._line + '\n') >> File "/home/leszek/lilypond/usr/lib/python2.4/codecs.py", line 501, in >> write >> return self.writer.write(data) >> File "/home/leszek/lilypond/usr/lib/python2.4/codecs.py", line 178, in >> write >> data, consumed = self.encode(object, self.errors) >> UnicodeDecodeError: 'utf8' codec can't decode byte 0xa9 in position >> 19: unexpected code byte >> >> >> (Isn't '0xa9' the copyright character? ;)) >> >> The XML file, if someone wants to take a peek, is here: >> https://www.dropbox.com/s/1dnv1hthyp7ss5h/Kyrie.xml?dl=0 >> >> Unfortunately it seems I do not have an XML file about which I could >> be sure that it converted correctly using a version of Lilypond newer >> than 2.19.5. Anyway, since one and the same XML file stopped >> converting correctly, it indeed seems that in the meantime some >> encoding issues have been introduced to musicxml2ly (I'm using 2.19.48 >> now). >> >> If you think I could be of any additional help (well, not sure this >> was helpful at all, but I'm trying ;)), please tell me what info I >> should provide. >> >> Cheers and thank you again, >> >> Leszek. >> >> ___ >> lilypond-user mailing list >> lilypond-user@gnu.org >> https://lists.gnu.org/mailman/listinfo/lilypond-user >> >> ___ lilypond-user mailing list lilypond-user@gnu.org
Re: XML to .ly and Lilypond, again
On Sun, 14 May 2017 16:06:47 +0200, Urs Liskawrote: > But can that be? Shouldn't MusicXML allow arbitrary regular Unicode > characters? Yes, but the file should be encoded using a single encoding. In this case, some parts of the file are UTF-8 encoded while other parts are UTF-16. ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
- Original Message - From: "Urs Liska" <u...@openlilylib.org> To: <lilypond-user@gnu.org> Sent: Sunday, May 14, 2017 3:06 PM Subject: Re: XML to .ly and Lilypond, again Am 14.05.2017 um 16:03 schrieb Phil Holmes: I've just confirmed Ian Ring's suggestion - removing the copyright symbol allows the conversion to continue, but results in text with spurious null characters. But can that be? Shouldn't MusicXML allow arbitrary regular Unicode characters? Urs My understanding is the XML is like HTML and requires special characters to be escaped. -- Phil Holmes ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
Am 14.05.2017 um 16:03 schrieb Phil Holmes: > I've just confirmed Ian Ring's suggestion - removing the copyright > symbol allows the conversion to continue, but results in text with > spurious null characters. But can that be? Shouldn't MusicXML allow arbitrary regular Unicode characters? Urs -- u...@openlilylib.org https://openlilylib.org http://lilypondblog.org ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
- Original Message - From: "Phil Holmes" <m...@philholmes.net> To: "Leszek Wronski" <elw...@gmail.com>; <lilypond-user@gnu.org> Sent: Sunday, May 14, 2017 2:31 PM Subject: Re: XML to .ly and Lilypond, again - Original Message - From: "Leszek Wronski" <elw...@gmail.com> To: <lilypond-user@gnu.org> Sent: Friday, May 12, 2017 12:54 PM Subject: Re: XML to .ly and Lilypond, again Dear Phil, I read through the 2 pages of comments at the link you provided. Is the ultimate suggestion the one in your last comment, that is, "I believe this patch requires Python 2.7, and we ship 2.4. As a result, windows users can no longer use musicxml2ly." ? If so, I should just mention that I'm getting the errors on an Ubuntu 16.04 system with Python version 2.7.12. Cheers, Leszek. OK - I've done some more looking at this. Using a test file produced by Sibelius, I get no problems using musicxml2ly on Ubuntu 14.04, Python 2.7.6. I think it would be helpful if people with other Ubuntu/Pytho versions could check an XML file (I can email my test to anyone who can check) to see whether the incorrect encoding of text is happening. If I try the same file on Windows, with LilyPond 2.19.43, all is well. With 2.19.44 I get the same errors as Leszek: Traceback (most recent call last): File "C:\Program Files (x86)\LilyPondV2.19.44\usr\bin\musicxml2ly.py", line 3269, in ? main() File "C:\Program Files (x86)\LilyPondV2.19.44\usr\bin\musicxml2ly.py", line 3263, in main voices = convert(filename, options) File "C:\Program Files (x86)\LilyPondV2.19.44\usr\bin\musicxml2ly.py", line 3175, in convert score_information.print_ly(printer) File "out/musicexp.py", line 855, in print_ly File "out/musicexp.py", line 848, in format_header_strings File "out/musicexp.py", line 123, in newline File "C:\Program Files (x86)\LilyPondV2.19.43\usr\lib\python2.4\codecs.py", line 501, in write return self.writer.write(data) File "C:\Program Files (x86)\LilyPondV2.19.43\usr\lib\python2.4\codecs.py", line 178, in write data, consumed = self.encode(object, self.errors) UnicodeDecodeError: 'utf8' codec can't decode byte 0xa9 in position 39: unexpected code byte However, if I try a file with a simpler header (e.g. 02d-Rests-Multimeasure-TimeSignatures.xml in the regtest suite), then the script runs successfully, but I get an output file with spurious nulls in text fields: \header { texidoc = "Multi-Measure rests should always be converted into durations that are a multiple of the time signature." } So it looks clear that something probably introduced in https://sourceforge.net/p/testlilyissues/issues/4751/ is causing encoding problems on Windows. As I say, I can't reproduce the problem on Linux. -- Phil Holmes I've just confirmed Ian Ring's suggestion - removing the copyright symbol allows the conversion to continue, but results in text with spurious null characters. -- Phil Holmes ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
- Original Message - From: "Leszek Wronski" <elw...@gmail.com> To: <lilypond-user@gnu.org> Sent: Friday, May 12, 2017 12:54 PM Subject: Re: XML to .ly and Lilypond, again Dear Phil, I read through the 2 pages of comments at the link you provided. Is the ultimate suggestion the one in your last comment, that is, "I believe this patch requires Python 2.7, and we ship 2.4. As a result, windows users can no longer use musicxml2ly." ? If so, I should just mention that I'm getting the errors on an Ubuntu 16.04 system with Python version 2.7.12. Cheers, Leszek. OK - I've done some more looking at this. Using a test file produced by Sibelius, I get no problems using musicxml2ly on Ubuntu 14.04, Python 2.7.6. I think it would be helpful if people with other Ubuntu/Pytho versions could check an XML file (I can email my test to anyone who can check) to see whether the incorrect encoding of text is happening. If I try the same file on Windows, with LilyPond 2.19.43, all is well. With 2.19.44 I get the same errors as Leszek: Traceback (most recent call last): File "C:\Program Files (x86)\LilyPondV2.19.44\usr\bin\musicxml2ly.py", line 3269, in ? main() File "C:\Program Files (x86)\LilyPondV2.19.44\usr\bin\musicxml2ly.py", line 3263, in main voices = convert(filename, options) File "C:\Program Files (x86)\LilyPondV2.19.44\usr\bin\musicxml2ly.py", line 3175, in convert score_information.print_ly(printer) File "out/musicexp.py", line 855, in print_ly File "out/musicexp.py", line 848, in format_header_strings File "out/musicexp.py", line 123, in newline File "C:\Program Files (x86)\LilyPondV2.19.43\usr\lib\python2.4\codecs.py", line 501, in write return self.writer.write(data) File "C:\Program Files (x86)\LilyPondV2.19.43\usr\lib\python2.4\codecs.py", line 178, in write data, consumed = self.encode(object, self.errors) UnicodeDecodeError: 'utf8' codec can't decode byte 0xa9 in position 39: unexpected code byte However, if I try a file with a simpler header (e.g. 02d-Rests-Multimeasure-TimeSignatures.xml in the regtest suite), then the script runs successfully, but I get an output file with spurious nulls in text fields: \header { texidoc = "Multi-Measure rests should always be converted into durations that are a multiple of the time signature." } So it looks clear that something probably introduced in https://sourceforge.net/p/testlilyissues/issues/4751/ is causing encoding problems on Windows. As I say, I can't reproduce the problem on Linux. -- Phil Holmes ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
Hi David, Ref.: "What's the role of TeXShop? What process did it carry out? " TeXShop, by means of its LilyPond Engine, processed the file containing the code copied from Frescobaldi. I processed the file with two LilyPond versions, 2.18.2 and 2.19.58.1, and as far I could see the results were identical. I always use TeXShop to edit LilyPond-files because it allows me to use some tricks and shortcuts that Frescobaldi does not like, or does not allow. As TeXShop always saves files with the extension .tex I rename the copies (_ly.tex). So the pdf in the package I uploaded was made with TeXShop. Incidentally, It is not really necessary to make a copy of the code generated by Frescobaldi from the xml-file. I can open and process the .ly -file with TeXShop just as well. But for most of the scores I make I have to do quite a lot of editing for which I always use TeXShop. Ref.: "The 2.19 files might be of more interest as 2.18.2 doesn't have the problem." There is no separate and/or different 2.19 file. The .ly -file generated from the downloaded -xml-file produced the same result with 2.18.2 and 2.19.58.1. Best regards, Robert On 14 May 2017, at 14:24 , David Wrightwrote: > On Sun 14 May 2017 at 12:42:03 (+0200), Robert Blackstone wrote: >> Hi Leszek, >> >> Out of curiosity I downloaded and processed the Kyrie.xml, from the link you >> attached to your post of 12-05-17. >> I was curious in particular because I mostly use the same procedure for >> engraving with LilyPond as you do, starting with Finale, except that I'm on >> a Mac (OSX 10.9.5 also known as Mavericks). >> So I imported your file in Frescobaldi, and then copied and pasted it in >> TeXShop, my favourite editor, which processed your file without any problem >> (both with LilyPond version 2.18.2 and 2.19.58.1). > > So with Frescobaldi you import the XML (which uses the musicxml2ly > program; you posted the .ly) and then produce the PDF (posted). > What's the role of TeXShop? What process did it carry out? > >> You can find the results here: >> >> https://dl.dropboxusercontent.com/u/512573/FileChute/LilyPond-test_Kyrie_from_xml-file.zip > > The 2.19 files might be of more interest as 2.18.2 doesn't have the problem. > >> So apparently for a Mac there is nothing wrong, neither with your xml-file >> nor with recent musicxml2ly. >> It may be of litte help to you (as long as you stay on Windows ;-)) but it >> could perhaps help you in narrowing down the search for the source of >> trouble in your system. > >> On 12 May 2017, at 13:42 , Leszek Wroński wrote: >>> […] Anyway, since one and the same XML file stopped >>> converting correctly, it indeed seems that in the meantime some >>> encoding issues have been introduced to musicxml2ly (I'm using 2.19.48 >>> now). > > Cheers, > David. ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
On Sun 14 May 2017 at 12:42:03 (+0200), Robert Blackstone wrote: > Hi Leszek, > > Out of curiosity I downloaded and processed the Kyrie.xml, from the link you > attached to your post of 12-05-17. > I was curious in particular because I mostly use the same procedure for > engraving with LilyPond as you do, starting with Finale, except that I'm on a > Mac (OSX 10.9.5 also known as Mavericks). > So I imported your file in Frescobaldi, and then copied and pasted it in > TeXShop, my favourite editor, which processed your file without any problem > (both with LilyPond version 2.18.2 and 2.19.58.1). So with Frescobaldi you import the XML (which uses the musicxml2ly program; you posted the .ly) and then produce the PDF (posted). What's the role of TeXShop? What process did it carry out? > You can find the results here: > > https://dl.dropboxusercontent.com/u/512573/FileChute/LilyPond-test_Kyrie_from_xml-file.zip The 2.19 files might be of more interest as 2.18.2 doesn't have the problem. > So apparently for a Mac there is nothing wrong, neither with your xml-file > nor with recent musicxml2ly. > It may be of litte help to you (as long as you stay on Windows ;-)) but it > could perhaps help you in narrowing down the search for the source of trouble > in your system. > On 12 May 2017, at 13:42 , Leszek Wrońskiwrote: > > […] Anyway, since one and the same XML file stopped > > converting correctly, it indeed seems that in the meantime some > > encoding issues have been introduced to musicxml2ly (I'm using 2.19.48 > > now). Cheers, David. ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
Hi Leszek, Out of curiosity I downloaded and processed the Kyrie.xml, from the link you attached to your post of 12-05-17. I was curious in particular because I mostly use the same procedure for engraving with LilyPond as you do, starting with Finale, except that I'm on a Mac (OSX 10.9.5 also known as Mavericks). So I imported your file in Frescobaldi, and then copied and pasted it in TeXShop, my favourite editor, which processed your file without any problem (both with LilyPond version 2.18.2 and 2.19.58.1). You can find the results here: https://dl.dropboxusercontent.com/u/512573/FileChute/LilyPond-test_Kyrie_from_xml-file.zip So apparently for a Mac there is nothing wrong, neither with your xml-file nor with recent musicxml2ly. It may be of litte help to you (as long as you stay on Windows ;-)) but it could perhaps help you in narrowing down the search for the source of trouble in your system. Best regards Robert Blackstone On 12 May 2017, at 13:42 , Leszek Wrońskiwrote: > Dear Guys, > > first, thank you very much for the replies! > > Second, for a little bit of detective work. I have not updated my > Finale in years, so if it produced correct XML files in the past, it > should do the same now (I always write using a similar really basic > setup, since actual editing is to be done in Lilypond). Anyway, I > tracked down an XML file which converted correctly in 2.19.5. I wanted > to check whether I would get the 'lots of null characters' problem > currently. To my surprise, I didn't even get to that part. Musicxml2ly > gave me the following: > > -- > musicxml2ly: Reading MusicXML from Kyrie.xml ... > musicxml2ly: Converting to LilyPond expressions... > (.) > musicxml2ly: Output to `Kyrie.ly' > Traceback (most recent call last): > File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3269, in ? >main() > File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3263, in main >voices = convert(filename, options) > File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3175, in convert >score_information.print_ly(printer) > File "/home/leszek/lilypond/usr/share/lilypond/current/python/musicexp.py", > line 855, in print_ly >self.format_header_strings(k, v, printer) > File "/home/leszek/lilypond/usr/share/lilypond/current/python/musicexp.py", > line 848, in format_header_strings >printer.newline() > File "/home/leszek/lilypond/usr/share/lilypond/current/python/musicexp.py", > line 123, in newline >self._file.write (self._line + '\n') > File "/home/leszek/lilypond/usr/lib/python2.4/codecs.py", line 501, in write >return self.writer.write(data) > File "/home/leszek/lilypond/usr/lib/python2.4/codecs.py", line 178, in write >data, consumed = self.encode(object, self.errors) > UnicodeDecodeError: 'utf8' codec can't decode byte 0xa9 in position > 19: unexpected code byte > > > (Isn't '0xa9' the copyright character? ;)) > > The XML file, if someone wants to take a peek, is here: > https://www.dropbox.com/s/1dnv1hthyp7ss5h/Kyrie.xml?dl=0 > > Unfortunately it seems I do not have an XML file about which I could > be sure that it converted correctly using a version of Lilypond newer > than 2.19.5. Anyway, since one and the same XML file stopped > converting correctly, it indeed seems that in the meantime some > encoding issues have been introduced to musicxml2ly (I'm using 2.19.48 > now). > > If you think I could be of any additional help (well, not sure this > was helpful at all, but I'm trying ;)), please tell me what info I > should provide. > > Cheers and thank you again, > > Leszek. > > ___ > lilypond-user mailing list > lilypond-user@gnu.org > https://lists.gnu.org/mailman/listinfo/lilypond-user ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
>On Fri 12 May 2017 at 12:03:36 (+0200), David Kastrup wrote: > >My guess is that the wrong encoding comes from "original" strings >being passed from the input to the output as opposed to all the >normal strings that are generated by the conversion program. These >original strings would carry their encoding with them. > > If you are meaning finale is producing the wrong xml, I can tell you Sibelius, musescore and capella also have this problem. ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
On Fri 12 May 2017 at 12:03:36 (+0200), David Kastrup wrote: > Urs Liskawrites: > > > Am 12.05.2017 um 11:26 schrieb Jan-Peter Voigt: > >> Dear Leszek, > >> > >> when I look with hexdump, I see a lot of NUL-bytes inside the file. > >> And it seems to me that the strings are in UTF-16 and the rest in > >> latin-1 or the like. At least it seems like a mixture of encodings. > > > > I think this is exactly what was discussed in the mentioned recent thread. > > So it seems this is an issue in musicxml2ly that has recently been > > introduced. Does anyone know or can anyone find out when this happened > > (by trying musicxml2ly from different LilyPond versions)? > > My guess would rather be on some auto-encoding/decoding choice by > Python, possibly triggered by badly (or unexpectedly?) encoded material > in the MusicXML file. My observations are: . The downloaded XML file has probably come from Windows as it has CR-LF line terminations. . musicxml2ly 2.18.2 handles the file just fine. . musicxml2ly 2.19.49 can't handle writing © to the output. . With the © replaced by C in the XML file, 2.19.49 can process the file, but some of the output is encoded wrongly (UTF16?). . Strings that might sometimes be expected to contain non-US-ASCII are the ones mainly affected, even where the string here happens to be all US-ASCII (Vivace, Allegro). . It's noticeable that the only other parts of this file encoded badly are \key commands and the final \midi and \tempo commands. . None of the strings introduced by = is affected. My guess is that the wrong encoding comes from "original" strings being passed from the input to the output as opposed to all the normal strings that are generated by the conversion program. These original strings would carry their encoding with them. Things that would fit into that model are the titles etc (I suspect the way the credit_dict dictionary is handled), the names of the (musical) keys (perhaps surviving a change from upper to lower case), and the midi tempo, "92". Cheers, David. ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
- Original Message - From: "Leszek Wronski" <elw...@gmail.com> To: <lilypond-user@gnu.org> Sent: Friday, May 12, 2017 12:54 PM Subject: Re: XML to .ly and Lilypond, again Dear Phil, I read through the 2 pages of comments at the link you provided. Is the ultimate suggestion the one in your last comment, that is, "I believe this patch requires Python 2.7, and we ship 2.4. As a result, windows users can no longer use musicxml2ly." ? If so, I should just mention that I'm getting the errors on an Ubuntu 16.04 system with Python version 2.7.12. Cheers, Leszek. In a way, that's good news. Much easier to debug on Linux. -- Phil Holmes ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
Dear Phil, I read through the 2 pages of comments at the link you provided. Is the ultimate suggestion the one in your last comment, that is, "I believe this patch requires Python 2.7, and we ship 2.4. As a result, windows users can no longer use musicxml2ly." ? If so, I should just mention that I'm getting the errors on an Ubuntu 16.04 system with Python version 2.7.12. Cheers, Leszek. - - Original Message - From: "Urs Liska" I think this is exactly what was discussed in the mentioned recent thread. So it seems this is an issue in musicxml2ly that has recently been introduced. Does anyone know or can anyone find out when this happened (by trying musicxml2ly from different LilyPond versions)? Urs See https://sourceforge.net/p/testlilyissues/issues/4751/ -- Phil Holmes ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
Dear Guys, first, thank you very much for the replies! Second, for a little bit of detective work. I have not updated my Finale in years, so if it produced correct XML files in the past, it should do the same now (I always write using a similar really basic setup, since actual editing is to be done in Lilypond). Anyway, I tracked down an XML file which converted correctly in 2.19.5. I wanted to check whether I would get the 'lots of null characters' problem currently. To my surprise, I didn't even get to that part. Musicxml2ly gave me the following: -- musicxml2ly: Reading MusicXML from Kyrie.xml ... musicxml2ly: Converting to LilyPond expressions... (.) musicxml2ly: Output to `Kyrie.ly' Traceback (most recent call last): File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3269, in ? main() File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3263, in main voices = convert(filename, options) File "/home/leszek/lilypond/usr/bin/musicxml2ly", line 3175, in convert score_information.print_ly(printer) File "/home/leszek/lilypond/usr/share/lilypond/current/python/musicexp.py", line 855, in print_ly self.format_header_strings(k, v, printer) File "/home/leszek/lilypond/usr/share/lilypond/current/python/musicexp.py", line 848, in format_header_strings printer.newline() File "/home/leszek/lilypond/usr/share/lilypond/current/python/musicexp.py", line 123, in newline self._file.write (self._line + '\n') File "/home/leszek/lilypond/usr/lib/python2.4/codecs.py", line 501, in write return self.writer.write(data) File "/home/leszek/lilypond/usr/lib/python2.4/codecs.py", line 178, in write data, consumed = self.encode(object, self.errors) UnicodeDecodeError: 'utf8' codec can't decode byte 0xa9 in position 19: unexpected code byte (Isn't '0xa9' the copyright character? ;)) The XML file, if someone wants to take a peek, is here: https://www.dropbox.com/s/1dnv1hthyp7ss5h/Kyrie.xml?dl=0 Unfortunately it seems I do not have an XML file about which I could be sure that it converted correctly using a version of Lilypond newer than 2.19.5. Anyway, since one and the same XML file stopped converting correctly, it indeed seems that in the meantime some encoding issues have been introduced to musicxml2ly (I'm using 2.19.48 now). If you think I could be of any additional help (well, not sure this was helpful at all, but I'm trying ;)), please tell me what info I should provide. Cheers and thank you again, Leszek. ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
- Original Message - From: "Urs Liska" <u...@openlilylib.org> To: <lilypond-user@gnu.org>; <bug-lilyp...@gnu.org> Sent: Friday, May 12, 2017 11:00 AM Subject: Re: XML to .ly and Lilypond, again Am 12.05.2017 um 11:26 schrieb Jan-Peter Voigt: Dear Leszek, when I look with hexdump, I see a lot of NUL-bytes inside the file. And it seems to me that the strings are in UTF-16 and the rest in latin-1 or the like. At least it seems like a mixture of encodings. I think this is exactly what was discussed in the mentioned recent thread. So it seems this is an issue in musicxml2ly that has recently been introduced. Does anyone know or can anyone find out when this happened (by trying musicxml2ly from different LilyPond versions)? Urs See https://sourceforge.net/p/testlilyissues/issues/4751/ -- Phil Holmes ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
Urs Liskawrites: > Am 12.05.2017 um 11:26 schrieb Jan-Peter Voigt: >> Dear Leszek, >> >> when I look with hexdump, I see a lot of NUL-bytes inside the file. >> And it seems to me that the strings are in UTF-16 and the rest in >> latin-1 or the like. At least it seems like a mixture of encodings. > > I think this is exactly what was discussed in the mentioned recent thread. > So it seems this is an issue in musicxml2ly that has recently been > introduced. Does anyone know or can anyone find out when this happened > (by trying musicxml2ly from different LilyPond versions)? My guess would rather be on some auto-encoding/decoding choice by Python, possibly triggered by badly (or unexpectedly?) encoded material in the MusicXML file. -- David Kastrup ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
Am 12.05.2017 um 11:26 schrieb Jan-Peter Voigt: > Dear Leszek, > > when I look with hexdump, I see a lot of NUL-bytes inside the file. > And it seems to me that the strings are in UTF-16 and the rest in > latin-1 or the like. At least it seems like a mixture of encodings. I think this is exactly what was discussed in the mentioned recent thread. So it seems this is an issue in musicxml2ly that has recently been introduced. Does anyone know or can anyone find out when this happened (by trying musicxml2ly from different LilyPond versions)? Urs > > Jan-Peter > > Am 12.05.2017 um 11:14 schrieb Leszek Wroński: >> Dear Guys, >> >> I see a similar issue has been discussed recently, so apologies for >> bringing it up again, but I want to highlight a different aspect of >> it, at least I think I do -- sorry if I'm mistaken. My point is that >> the .ly code converted from .xml does not compile simply under >> lilypond (I'm using 2.19.48), and not just 'in Frescobaldi'. >> >> Throughout the years my method of working has been the following: >> 1. Enter music in Finale, on Windows 7, via a MIDI keyboard >> 2. Export to MusicXML >> 3. Switch to Ubuntu Linux >> 3. Convert to .ly >> 4. Edit in Frescobaldi. >> >> I haven't done the converting in a while; I've been editing some older >> files. However, today I realised that my simple method is no longer >> working. I arrived at this file: >> https://www.dropbox.com/s/cmvbn3z4jsar613/Herr_nun_01.ly?dl=0 which >> has at the header the following: >> >> 5. \header { >> 6.encodingsoftware = >> 7."Finale 2012 for Windows" >> >> (...) >> >> and Lilypond itself, not just Frescobaldi, gives me this: >> >> --- >> GNU LilyPond 2.19.48 >> Processing `Herr_nun_01.ly' >> Parsing... >> Herr_nun_01.ly:7:5: error: EOF found inside string >> >> " >> Herr_nun_01.ly:7:5: error: syntax error, unexpected end of input, >> expecting \header >> >> " >> fatal error: failed files: "Herr_nun_01.ly" >> --- >> >> (Frescobaldi's message is similar.) >> >> Even after removing the header altogether, the error just 'moves down >> the file'. I am completely baffled by this. I looked at the file using >> a Hex viewer and I see nothing wrong with it. My old files compile >> correctly, so it seems something has to be wrong with the new file. >> >> Am I making some stupid mistake? Could anyone offer some advice? >> >> Best regards, >> >> Leszek. >> >> ___ >> lilypond-user mailing list >> lilypond-user@gnu.org >> https://lists.gnu.org/mailman/listinfo/lilypond-user >> > > > ___ > lilypond-user mailing list > lilypond-user@gnu.org > https://lists.gnu.org/mailman/listinfo/lilypond-user -- u...@openlilylib.org https://openlilylib.org http://lilypondblog.org ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
Guys, I was a bit too quick with 'It all looks OK in Hex". When I opend the .ly in Emacs I saw "^@" characters after all letters ("t^@i^@m^@e^@" etc). I removed all of them. This almost did the trick and finally the file was compilable after a few minor edits. So it seems that the 'EOF' issue is present under Linux, too, but can be easily overcome. Cheers, Leszek. ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
On Fri, 12 May 2017 11:14:53 +0200, Leszek Wrońskiwrote: > I looked at the file using > a Hex viewer and I see nothing wrong with it. The strings like "Finale ..." and some other parts of the file are multy-byte (UTF-16) encoded while the rest of the file is single-byte, so LilyPond interprets it as single-byte and gets lost completely. In Emacs, it looks like this (^@ denotes a 0 byte): \header { encodingsoftware = "^@F^@i^@n^@a^@l^@e^@ " but also: PartPOneVoiceOne = \relative g'' { \clef "treble" \^@k^@e^@y^@ ^@g^@ ^@\^@m^@i^@n^@o^@r... -- Johan ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user
Re: XML to .ly and Lilypond, again
Dear Leszek, when I look with hexdump, I see a lot of NUL-bytes inside the file. And it seems to me that the strings are in UTF-16 and the rest in latin-1 or the like. At least it seems like a mixture of encodings. Jan-Peter Am 12.05.2017 um 11:14 schrieb Leszek Wroński: Dear Guys, I see a similar issue has been discussed recently, so apologies for bringing it up again, but I want to highlight a different aspect of it, at least I think I do -- sorry if I'm mistaken. My point is that the .ly code converted from .xml does not compile simply under lilypond (I'm using 2.19.48), and not just 'in Frescobaldi'. Throughout the years my method of working has been the following: 1. Enter music in Finale, on Windows 7, via a MIDI keyboard 2. Export to MusicXML 3. Switch to Ubuntu Linux 3. Convert to .ly 4. Edit in Frescobaldi. I haven't done the converting in a while; I've been editing some older files. However, today I realised that my simple method is no longer working. I arrived at this file: https://www.dropbox.com/s/cmvbn3z4jsar613/Herr_nun_01.ly?dl=0 which has at the header the following: 5. \header { 6.encodingsoftware = 7."Finale 2012 for Windows" (...) and Lilypond itself, not just Frescobaldi, gives me this: --- GNU LilyPond 2.19.48 Processing `Herr_nun_01.ly' Parsing... Herr_nun_01.ly:7:5: error: EOF found inside string " Herr_nun_01.ly:7:5: error: syntax error, unexpected end of input, expecting \header " fatal error: failed files: "Herr_nun_01.ly" --- (Frescobaldi's message is similar.) Even after removing the header altogether, the error just 'moves down the file'. I am completely baffled by this. I looked at the file using a Hex viewer and I see nothing wrong with it. My old files compile correctly, so it seems something has to be wrong with the new file. Am I making some stupid mistake? Could anyone offer some advice? Best regards, Leszek. ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user ___ lilypond-user mailing list lilypond-user@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-user