Re: guile-2.0 and debian

[Paul]

On 11/25/2016 06:13 PM, Thomas Morley wrote:


- Investigate why lilypond is a lot slower when using guile-2.0

This is the current main problem, I'd say.


From what I've read, the guile 2.0 interpreter is simply slower than 
the guile 1.8 interpreter (with the trade-off being that 2.0 has a 
compiler).  For example, a quote from:


https://wingolog.org/archives/2016/01/11/the-half-strap-self-hosting-and-guile

-begin-quote

Back in 2009 when we switched to the eval-in-Scheme, we knew that it 
would result in a slower interpreter. This is because instead of the 
interpreter being compiled to native code, it was compiled to bytecode. 
Also, Guile's Scheme compiler wasn't as good then, so we knew that we 
were leaving optimizations on the floor. Still, the switch to an 
evaluator in Scheme enabled integration of the compiler, and we thought 
that the interpreter speed would improve with time. I just took a look 
and with this silly loop:


(let lp ((n 0)) (if (< n #e1e7) (lp (1+ n

Guile 1.8's interpreter written in C manages to run this in 1.1 seconds. 
Guile 2.0's interpreter written in Scheme and compiled to the old 
virtual machine does it in 16.4 seconds. Guile 2.1.1's interpreter, with 
the closure-chaining optimization, a couple of peephole optimizations in 
the interpreter, and compiled using the better compiler and VM from 
Guile 2.2, manages to finish in 2.4 seconds. So we are definitely 
getting better, and by the time we compile eval.scm to native code I 
have no doubt that we will be as good as the old C implementation. (Of 
course, when compiled to Guile 2.2's VM, the loop finishes in 55 
/milli/seconds, but comparing a compiler and an interpreter is no fair.)


--end-quote

So I assume (as a non-authority in this area) that the best way to 
address the slowdown is to start using guile 2.0's compiler to compile 
LilyPond's scheme code.  And I suppose taking that step would make sense 
after the known bugs are fixed and things work using the interpreter, 
but are just slower.


-Paul



___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: guile-2.0 and debian

[Thomas Morley]

2016-11-25 23:19 GMT+01:00 Antonio Ospite <a...@ao2.it>:
> On Fri, 25 Nov 2016 09:33:42 +0100
> David Kastrup <d...@gnu.org> wrote:
>
>> David Kastrup <d...@gnu.org> writes:
>>
>> > Thomas Morley <thomasmorle...@gmail.com> writes:
>> >
> [...]
>> >> with the container and/or LilyDev for guile2 we are going to have some
>> >> stuff to simplify the workflow, attracting more people.
>> >> It's still tedious to add Antonios patches.
>> >> How about creating a public branch?
>> >> I could do this in the evening (if I remember the syntax)

Too tired to do it today anyway...

>> >>
>> >> What do you think?
>> >
>> > Shouldn't most of them be able to become part of master?
>>
>
> TBH I didn't try to compile lilypond using guile-1.8 with my patches
> applied, I can give it a go in the next days.
>
>> Though the time frame of master moves in weeks, and the impetus of
>> developer activity may move in hours.
>>
>> So a separate branch for communication might indeed be a reasonable
>> short-term measure.
>>
>
> I agree, but since these patches are a moving target we should choose
> one of these two alternative approaches:
>
>  1. Use only one branch, but warn users that the branch might be
> rebased, this would mean that "git pull --force" might be needed on
> the client side.
>
>  2. Use versioned branches, this means that when the need for a rebase
> arises the result of the rebasing goes into a new branch, e.g.:
>
>   guile-2.0-v2
>
> or
>
>   guile-2.0-20161125
>
>
> In the mean time I started to write a TODO list of the missing pieces:
>
> - Make lilypond locale-independent
>
> - /input/regression/keys.ly looks bad (Reported by Thomas Morley)

This is due to the fix for
https://sourceforge.net/p/testlilyissues/issues/660/

Not related with guile2, so I'd drop it from _this_ TODO-list.

> - /input/regression/utf-8.ly changed spacing/line.break (Reported by
>   Thomas Morley) Might be related to the floating point issue below
>
> - Investigate why the floating point numbers are different in some
>   decimal digits (Reported by Knut Petersen)
>
> - Investigate why lilypond is a lot slower when using guile-2.0

This is the current main problem, I'd say.

Probably worth adding:

(display (make-simple-markup "xy"))

results in:

2.19.51:
(# xy)

2.19.52-guile2
(# xy)
Which is technically correct but not helpful.


Cheers,
  Harm

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: guile-2.0 and debian

[Knut Petersen]

Hi Antonio!

TBH I didn't try to compile lilypond using guile-1.8 with my patches
applied, I can give it a go in the next days.

I added your patches in numerical order as single commits to my local branch on 
top
of current master and started a "git bisect run" process to investigate which 
of your
patches breaks building lilypond with guile 1.8 with the following result:

   530823b891f732293ab4b6311b0ca96b1e863c88 is the first bad commit
   commit 530823b891f732293ab4b6311b0ca96b1e863c88
   Author: Antonio Ospite 
   Date:   Wed Oct 26 16:42:32 2016 +0200

Update changes from commit 122525f (Keep GUILEv2 from redecoding string 
input)

Update the changes from commit 122525f (Source_file::init_port: Keep
GUILEv2 from redecoding string input) to follow what was done in commit
652f454 (Add lily/lily-modules.cc).



cu,
 Knut
___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: lilypond, guile 2 and OpenSuSE

[Antonio Ospite]

On Fri, 25 Nov 2016 13:29:39 +0100
Knut Petersen  wrote:

> Hi everybody!
> 
> Status of guile on OpenSuSE distributions:
> ==
[...]
> Building current lilypond master + the 15 patches from
> ao2.it/tmp/lilypond-guile2/patches_2016-11-22 using guile 2 on
> Tumbleweed succeeds too:
[...]

Thanks for the test Knut.

> Tests
> =
> Compiling some of my own scores with both the guile 1 and guile
> 2 versions succeeds. The pdfs are visually identical, although they
> are not byte-identical. Compiling to postscript and comparing the
> results shows the reason:
> 
> knut@golem:~/Invention> diff Distler_LobeDenHerren_Schwabacher_g1.ps 
> Distler_LobeDenHerren_Schwabacher_g2.ps
[...]

> So there is no real problem, only an irrelevant difference in
> precision.
>

I noticed that too, but I haven't got the chance to investigate.

Another limitation is that lilypond (at least with guile-2) is dependent
on the user locale, and requires an UTF-8 locale to work in an
acceptable way.

> Speed
> =
> 
> Compiling a moderately complex score with both versions showed
> that the guile-2-version is much slower (real exec times):
> 
> guile-2-version: 0m20,032s 0m19,186 0m19,124s
> guile-1-version: 0m8,814s 0m9,023s 0m8,971s
>

I started to keep a TODO of what is left to make lilypond with guile-2
comparable with lilypond with guile-1.8, investigating where the
slowdown is one item in the list.

Thanks,
   Antonio

-- 
Antonio Ospite
https://ao2.it
https://twitter.com/ao2it

A: Because it messes up the order in which people normally read text.
   See http://en.wikipedia.org/wiki/Posting_style
Q: Why is top-posting such a bad thing?

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: guile-2.0 and debian

[Knut Petersen]

Am 25.11.2016 um 23:19 schrieb Antonio Ospite:



stuff to simplify the workflow, attracting more people.
It's still tedious to add Antonios patches.
How about creating a public branch?
I could do this in the evening (if I remember the syntax)

What do you think?

Shouldn't most of them be able to become part of master?

TBH I didn't try to compile lilypond using guile-1.8 with my patches
applied, I can give it a go in the next days.

I tried it this afternoon, it fails. But that should be easy solve.

cu,
 Knut

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: guile-2.0 and debian

[Antonio Ospite]

On Fri, 25 Nov 2016 09:33:42 +0100
David Kastrup <d...@gnu.org> wrote:

> David Kastrup <d...@gnu.org> writes:
> 
> > Thomas Morley <thomasmorle...@gmail.com> writes:
> >
[...]
> >> with the container and/or LilyDev for guile2 we are going to have some
> >> stuff to simplify the workflow, attracting more people.
> >> It's still tedious to add Antonios patches.
> >> How about creating a public branch?
> >> I could do this in the evening (if I remember the syntax)
> >>
> >> What do you think?
> >
> > Shouldn't most of them be able to become part of master?
>

TBH I didn't try to compile lilypond using guile-1.8 with my patches
applied, I can give it a go in the next days.

> Though the time frame of master moves in weeks, and the impetus of
> developer activity may move in hours.
> 
> So a separate branch for communication might indeed be a reasonable
> short-term measure.
>

I agree, but since these patches are a moving target we should choose
one of these two alternative approaches:

 1. Use only one branch, but warn users that the branch might be
rebased, this would mean that "git pull --force" might be needed on
the client side.

 2. Use versioned branches, this means that when the need for a rebase
arises the result of the rebasing goes into a new branch, e.g.:

  guile-2.0-v2

or

  guile-2.0-20161125


In the mean time I started to write a TODO list of the missing pieces:

- Make lilypond locale-independent

- /input/regression/keys.ly looks bad (Reported by Thomas Morley)

- /input/regression/utf-8.ly changed spacing/line.break (Reported by
  Thomas Morley) Might be related to the floating point issue below

- Investigate why the floating point numbers are different in some
  decimal digits (Reported by Knut Petersen)

- Investigate why lilypond is a lot slower when using guile-2.0


Thanks,
   Antonio

-- 
Antonio Ospite
https://ao2.it
https://twitter.com/ao2it

A: Because it messes up the order in which people normally read text.
   See http://en.wikipedia.org/wiki/Posting_style
Q: Why is top-posting such a bad thing?

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: instructions to create a Linux container (as replacement to LilyDev for Linux users)

[Federico Bruni]

I've just seen an error in my first email. This is correct:

 debootstrap --arch=amd64 stretch DebianStretch 
http://httpredir.debian.org/debian/

 sudo systemd-nspawn -D DebianStretch


of course you can replace DebianStretch with whatever directory name 
you prefer



___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: guile-2.0 and debian

[karl]

Jan-Peter Voigt:
> Am 24.11.2016 um 16:10 schrieb Antonio Ospite:
> > And about the choice between guile 1.8 or 2.0.12/13, it should be
> > possible to have both guile-1.8 and guile-2.0 packages installed in the
> > same image and then let lilypond choose which one to pick up at
> > configure-time, shoudln't it?
> I doubt that. You can install guile 1.8 and 2.0 in one system, but not 
> guile-dev 1.8 and 2.0 AFAICS. So one needs to separate systems for each 
> version of guile.
> 
> Or am I wrong?

I get this on a gentoo system:

# emerge lilypond
[...]
checking for guile-config... ./configure: line 7604: 11151 Aborted  
   $exe --version > /dev/null 2>&1
no
checking guile compile flags... ;;; note: source file 
/usr/share/guile/1.8/ice-9/boot-9.scm
;;;   newer than compiled /usr/local/lib/guile/2.0/ccache/ice-9/boot-9.go
Throw without catch before boot:
Throw to key syntax-error with args ("memoization" "In file ~S, line ~S: ~A 
~S." ("/usr/share/guile/1.8/ice-9/boot-9.scm" 101 "Bad define placement" 
(define (toplevel-env? env) (or (not (pair? env)) (not (pair? (car env)) 
#f)Aborting.

checking guile link flags... ;;; note: source file 
/usr/share/guile/1.8/ice-9/boot-9.scm
;;;   newer than compiled /usr/local/lib/guile/2.0/ccache/ice-9/boot-9.go
Throw without catch before boot:
Throw to key syntax-error with args ("memoization" "In file ~S, line ~S: ~A 
~S." ("/usr/share/guile/1.8/ice-9/boot-9.scm" 101 "Bad define placement" 
(define (toplevel-env? env) (or (not (pair? env)) (not (pair? (car env)) 
#f)Aborting.
[...]


# /usr/bin/guile-config --version 
guile-config - Guile version 1.8.8
# /usr/local/bin/guile-config --version
guile-config - Guile version 2.0.9

and putting v1.8.8 before v2.0.9 in the search patch doesn't help.

Regards,
/Karl Hammar

---
Aspö Data
Lilla Aspö 148
S-742 94 Östhammar
Sweden
+46 173 140 57



___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

instructions to create a Linux container (as replacement to LilyDev for Linux users)

[Federico Bruni]

Hi all

For those who want to contribute to the guile-2 migration, I suggest 
you use a Linux container, because I'm having troubles with building a 
new LilyDev image and I don't know when I'll find a solution.


I quickly wrote a brief tutorial to set up a Debian or Fedora guest 
within a systemd container (I guess that you need a recent distro).



# Linux container

## Create the guest system

### Debian

Install the following packages in your Debian-based host:

 sudo apt-get install systemd-container debootstrap

Then create the system (we'll use Debian testing, which has the 
required guile-2.0 package):


 mkdir debian
 debootstrap --arch=amd64 stretch DebianStretch 
http://httpredir.debian.org/debian/

 sudo systemd-nspawn -D debian

### Fedora

If you have Fedora, you can use this command:

 mkdir fedora
 sudo dnf --releasever=25 --installroot=/absolute/path/to/fedora 
install systemd passwd dnf fedora-release

 sudo systemd-nspawn -D fedora


## Guest setup

Whatever you chose, Debian or Fedora, let's continue from the last 
command.
The system will boot without asking for a password and you'll be the 
root user.

Let's set the root password and close:

 passwd
 shutdown -h now

Start it again with the -b option (now you'll have to login as root 
using

the password you chose):

 sudo systemd-nspawn -bD DebianStretch

Let's install LilyPond dependencies and some useful packages.
If you have Debian:

 apt-get install git nano
 apt-get build-dep lilypond

If you have Fedora:

 dnf install dnf-plugins-core git nano
 dnf builddep lilypond

Create a regular user for working on LilyPond source:

 adduser dev

Switch to dev user:

 su dev
 cd

Now you are in /home/dev in your guest. You can use your host file 
manager to

copy lilypond-git from your host to /home/dev/ in your guest.  Same for
~/.gitconfig, if you don't want to repeat the setup.

Use nano for git commit messages.
If you prefer, the LilyPond source files can be edited in the host 
system,

where you have also the GUI and your usual applications.

HTH
Federico




___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

lilypond, guile 2 and OpenSuSE

[Knut Petersen]

Hi everybody!

Status of guile on OpenSuSE distributions:
==

OpenSuSE Tumbleweed includes guile 2.0.13 since 2016-10-15.
OpenSuSE Leap does include guile 2, but the version is too old to support 
lilypond.
Manual installation of the guile 2 version from Tumbleweed is possible on Leap.
All current versions of the OpenSuSE distribution allow simultaneous 
installation of guile 1 and 2.

Building current lilypond master using guile 1 on Tumbleweed succeeds:
==

   Building lilypond 92d84685be9b496d4c983e555e846341ecc4e5f0... this will take 
some time ;-)
   configure flags to be used: --prefix=/home/knut/sources/lilyg1built/
   exec ./autogen.sh --noconfigure in /home/knut/sources/lily ... succeeded 
after 0 seconds
   exec ../configure --prefix=/home/knut/sources/lilyg1built/ in 
/home/knut/sources/lily/build ... succeeded after 5 seconds
   exec make -k -j 11 CPU_COUNT=11 all in /home/knut/sources/lily/build ... 
succeeded after 92 seconds
   exec make -j 11 CPU_COUNT=11 install in /home/knut/sources/lily/build ... 
succeeded after 2 seconds
   exec make -j 11 CPU_COUNT=11 doc in /home/knut/sources/lily/build ... 
succeeded after 831 seconds
   exec make -j 11 CPU_COUNT=11 install-doc in /home/knut/sources/lily/build 
... succeeded after 4 seconds
   Total lilypond build time: 934 seconds

Building current lilypond master + the 15 patches from 
ao2.it/tmp/lilypond-guile2/patches_2016-11-22
using guile 2 on Tumbleweed succeeds too:
==

   Building lilypond d0c4acca9fc6a9d9f2d2073ab01499809b7e4981... this will take 
some time ;-)
   configure flags to be used: --prefix=/home/knut/sources/lilyg2built/ 
--enable-guile2
   exec ./autogen.sh --noconfigure in /home/knut/sources/lilyg2 ... succeeded 
after 1 seconds
   exec ../configure --prefix=/home/knut/sources/lilyg2built/ --enable-guile2 
in /home/knut/sources/lilyg2/build ... succeeded after 5 seconds
   exec make -k -j 11 CPU_COUNT=11 all in /home/knut/sources/lilyg2/build ... 
succeeded after 98 seconds
   exec make -j 11 CPU_COUNT=11 install in /home/knut/sources/lilyg2/build ... 
succeeded after 1 seconds
   exec make -j 11 CPU_COUNT=11 doc in /home/knut/sources/lilyg2/build ... 
succeeded after 1313 seconds
   exec make -j 11 CPU_COUNT=11 install-doc in /home/knut/sources/lilyg2/build 
... succeeded after 5 seconds
   Total lilypond build time: 1423 seconds

Tests
=

Compiling some of my own scores with both the guile 1 and guile 2 versions 
succeeds. The pdfs are visually identical,
although they are not byte-identical. Compiling to postscript and comparing the 
results shows the reason:

   knut@golem:~/Invention> diff Distler_LobeDenHerren_Schwabacher_g1.ps 
Distler_LobeDenHerren_Schwabacher_g2.ps
   15c15
   < /lilypond-datadir where {pop} {userdict /lilypond-datadir 
(/home/knut/sources/lilyg1built/share/lilypond/2.19.52) put } ifelse
   ---
> /lilypond-datadir where {pop} {userdict /lilypond-datadir 
(/home/knut/sources/lilyg2built/share/lilypond/2.19.52) put } ifelse
   5639,5641c5639,5641
   < /lily-output-units 2.83464566929134 def %% millimeter
   < /staff-line-thickness 0.119078341013825 def
   < /line-width 124.824993649987 def
   ---
> /lily-output-units 2.834645669291339 def %% millimeter
> /staff-line-thickness 0.11907834101382486 def
> /line-width 124.82499364998729 def
   5644,5646c5644,5646
   < /output-scale 1.36190673861907 def
   < /page-height 218.076606553213 def
   < /page-width 154.195580391161 def
   ---
> /output-scale 1.3619067386190675 def
> /page-height 218.07660655321308 def
> /page-width 154.19558039116077 def


So there is no real problem, only an irrelevant difference in precision.

Speed
=

Compiling a moderately complex score with both versions showed
that the guile-2-version is much slower (real exec times):

   guile-2-version: 0m20,032s 0m19,186 0m19,124s
   guile-1-version: 0m8,814s 0m9,023s 0m8,971s

cu,
 Knut



___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: guile-2.0 and debian

[David Kastrup]

David Kastrup  writes:

> Thomas Morley  writes:
>
>> 2016-11-24 16:10 GMT+01:00 Antonio Ospite :
>>> On Thu, 24 Nov 2016 14:03:32 +0100
>>> Federico Bruni  wrote:
>>
>>
>>
>> Hi all,
>>
>> with the container and/or LilyDev for guile2 we are going to have some
>> stuff to simplify the workflow, attracting more people.
>> It's still tedious to add Antonios patches.
>> How about creating a public branch?
>> I could do this in the evening (if I remember the syntax)
>>
>> What do you think?
>
> Shouldn't most of them be able to become part of master?

Though the time frame of master moves in weeks, and the impetus of
developer activity may move in hours.

So a separate branch for communication might indeed be a reasonable
short-term measure.

-- 
David Kastrup

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: guile-2.0 and debian

[David Kastrup]

Thomas Morley  writes:

> 2016-11-24 16:10 GMT+01:00 Antonio Ospite :
>> On Thu, 24 Nov 2016 14:03:32 +0100
>> Federico Bruni  wrote:
>
>
>
> Hi all,
>
> with the container and/or LilyDev for guile2 we are going to have some
> stuff to simplify the workflow, attracting more people.
> It's still tedious to add Antonios patches.
> How about creating a public branch?
> I could do this in the evening (if I remember the syntax)
>
> What do you think?

Shouldn't most of them be able to become part of master?

-- 
David Kastrup

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel

Re: guile-2.0 and debian

[Thomas Morley]

2016-11-24 16:10 GMT+01:00 Antonio Ospite :
> On Thu, 24 Nov 2016 14:03:32 +0100
> Federico Bruni  wrote:



Hi all,

with the container and/or LilyDev for guile2 we are going to have some
stuff to simplify the workflow, attracting more people.
It's still tedious to add Antonios patches.
How about creating a public branch?
I could do this in the evening (if I remember the syntax)

What do you think?

Cheers,
  Harm

___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel