Re: Iterators in Scheme?

[Niols]

Hello,

On 07/08/2020 00:18, David Kastrup wrote:

Niols  writes:

I have not managed to override the "\repeat" music function.


It isn't a music function but a "reserved" word in LilyPond.


I know. Yet there is a "repeat" scheme function defined in 
scm/ly-syntax-constructors.ly that I tried to redefine (by recompiling 
LilyPond) and successfully managed to make it do what I wanted. I guess 
the lambda it contains gets passed to a table of some kind that 
associates reserved words to behaviours, but I could not locate such a 
table.



I suspect there is something I do not understand in the order in which
things are executed in LilyPond.

I have however found a solution based on overriding "make-music". I
replace "make-music" by a wrapper around it that adds the callback
from the snippet to all 'VoltaRepeatedMusic and leaves the rest
alone. This is actually just a few lines:

 #(define the-make-music make-music)

 #(define (make-music-wrapped name . music-properties)
   (let ((music (apply the-make-music (cons name music-properties
(if (equal? name 'VoltaRepeatedMusic)
 (ly:music-set-property! music 'elements-callback new-volta-set))
music))

 #(set! make-music make-music-wrapped)

It might not be the cleanest, but that is all I have, and that seems
to work.


It would be easier to do

music-descriptions.VoltaRepeatedMusic.elements-callback = ...

A cursory glance would suggest that might work (though its effect would
not be limited to one session).


Great idea! By reading scm/define-music-types.scm,I saw that 
music-descriptions is fed to a hash table. I thus managed this with:


#(let ((props (hashq-ref music-name-to-property-table 
'VoltaRepeatedMusic)))

  (assoc-set! props 'elements-callback new-volta-set))


You could also try to redefine the make-volta-set function.


That, I did not manage to make work. I suppose that the music-properties 
list and, later, the music-name-to-property-table only store the 
contents of make-volta-set; overriding it later is thus of no use.



2. Maybe there is a way to define a function that will be ran before
processing on any music? But this I have no idea if this is
possible, or how to do it, and my research has proven ineffective so
far.


There are the scorification hooks.


That sounds really interesting, whatever that is, also for other hacks 
around LilyPond. Would you happen to have any pointer for that?



3. Maybe there is a way to override the "make-volta-set" callback
instead of creating a new one? Such that then, in LilyPond's normal
execution, it uses the new definition and not and old one that then
needs to be replaced?


For 2. and 3., I still have no idea as this is far beyond my
understanding of the internals of LilyPond. I am not really looking
into it though.


Not a matter of internals.

#(define original-make-volta-set make-volta-set)

(define (make-volta-set ...) ...)

Possibly

(set! make-volta-set (lambda (...) ...))

but I think that the define probably works.


I suppose they work to override make-volta-set. As said earlier, though, 
I did not manage to get the expected result from this. I suppose the 
lambda is already fed to the music-name-to-property-table. Overriding 
this thus does not give much useful things.


Thank you very much for you help!
— Niols

Re: Iterators in Scheme?

[David Kastrup]

Niols  writes:

> Hello again,

[...]

> I have not managed to override the "\repeat" music function.

It isn't a music function but a "reserved" word in LilyPond.

> I suspect there is something I do not understand in the order in which
> things are executed in LilyPond.
>
> I have however found a solution based on overriding "make-music". I
> replace "make-music" by a wrapper around it that adds the callback
> from the snippet to all 'VoltaRepeatedMusic and leaves the rest
> alone. This is actually just a few lines:
>
> #(define the-make-music make-music)
>
> #(define (make-music-wrapped name . music-properties)
>   (let ((music (apply the-make-music (cons name music-properties
>(if (equal? name 'VoltaRepeatedMusic)
> (ly:music-set-property! music 'elements-callback new-volta-set))
>music))
>
> #(set! make-music make-music-wrapped)
>
> It might not be the cleanest, but that is all I have, and that seems
> to work.

It would be easier to do

music-descriptions.VoltaRepeatedMusic.elements-callback = ...

A cursory glance would suggest that might work (though its effect would
not be limited to one session).

You could also try to redefine the make-volta-set function.

>> 2. Maybe there is a way to define a function that will be ran before
>> processing on any music? But this I have no idea if this is
>> possible, or how to do it, and my research has proven ineffective so
>> far.

There are the scorification hooks.

>> 3. Maybe there is a way to override the "make-volta-set" callback 
>> instead of creating a new one? Such that then, in LilyPond's normal
>> execution, it uses the new definition and not and old one that then 
>> needs to be replaced?
>
> For 2. and 3., I still have no idea as this is far beyond my
> understanding of the internals of LilyPond. I am not really looking
> into it though.

Not a matter of internals.

#(define original-make-volta-set make-volta-set)

(define (make-volta-set ...) ...)

Possibly

(set! make-volta-set (lambda (...) ...))

but I think that the define probably works.


>
> Thank you for your help; have a great Summer!
> — Niols
>
>

-- 
David Kastrup

Re: Iterators in Scheme?

[Niols]

Hello again,

On 30/07/2020 18:22, Niols wrote:

On 29/07/2020 17:28, Jean-Julien Fleck wrote:
Le mer. 29 juil. 2020 à 15:44, Niols > a écrit :



 > What do you want to achieve?

    This is actually follow-up research from my side after a previous
    e-mail
    to this list. I would like to print bar numbers that are "repeat
    aware",
    that is such that if bars 9 to 12 repeat twice, then the bar 
numbers go


      1, 2, …, 9, …, 12, 17, 18, etc.

    with a jump after 12 that corresponds to the bars 13 to 16 that are
    actually 9 to 12. Even better, I would like to print the two bar
    numbers
    for these bars (9/13, 10/14, etc.)


Perhaps this snippet could be of some use for you: 
http://lsr.di.unimi.it/LSR/Item?id=1080


I have been playing with this solution and I find it very satisfactory 
for my use. I do not understand yet all the intrinsics (in particular 
with respect to the context-spec-music and the make-apply-context 
functions), but I understand roughly how it works.


There is just one thing that I would want to "improve" (that is, for my 
use): the solution needs to map on all the music in order to find all 
the VoltaRepeatedMusic and override their callbacks to a newly defined 
one. This is achieved by providing a function "unfoldBarNumbers" that 
can be applied to any music and that does it.


I would however be interested to have that for all the music in my file, 
without having to call that function. The different ideas that pop into 
my mind in such a situation are the following:


1. Maybe I could override the "\repeat" function so that it does the 
same thing as by default but, when it is a "volta" repeat, it also 
attaches the new callback to the created music. I believe I see how to 
achieve that, except I seem unsuccessful in overriding the default 
definition of "\repeat". Even if I do something like:


     #(define (repeat type num body alts)
   (display "\nrepeat\n"))

     { b4 \repeat volta 2 { b b } b }

it behaves as if I had not put the #(define ...) statement. I have 
played around this idea for a while without success. I have tried to see 
if "\repeat" had a special treatment (which it does, at least in the 
lexing/parsing), but I haven't been able to figure this out.


I have not managed to override the "\repeat" music function. I suspect 
there is something I do not understand in the order in which things are 
executed in LilyPond.


I have however found a solution based on overriding "make-music". I 
replace "make-music" by a wrapper around it that adds the callback from 
the snippet to all 'VoltaRepeatedMusic and leaves the rest alone. This 
is actually just a few lines:


#(define the-make-music make-music)

#(define (make-music-wrapped name . music-properties)
  (let ((music (apply the-make-music (cons name music-properties
   (if (equal? name 'VoltaRepeatedMusic)
(ly:music-set-property! music 'elements-callback new-volta-set))
   music))

#(set! make-music make-music-wrapped)

It might not be the cleanest, but that is all I have, and that seems to 
work.


2. Maybe there is a way to define a function that will be ran before 
processing on any music? But this I have no idea if this is possible, or 
how to do it, and my research has proven ineffective so far.


3. Maybe there is a way to override the "make-volta-set" callback 
instead of creating a new one? Such that then, in LilyPond's normal 
execution, it uses the new definition and not and old one that then 
needs to be replaced?


For 2. and 3., I still have no idea as this is far beyond my 
understanding of the internals of LilyPond. I am not really looking into 
it though.


Thank you for your help; have a great Summer!
— Niols

Re: Iterators in Scheme?

[Niols]

Hello again,


On 29/07/2020 17:28, Jean-Julien Fleck wrote:
Le mer. 29 juil. 2020 à 15:44, Niols > a écrit :



 > What do you want to achieve?

    This is actually follow-up research from my side after a previous
    e-mail
    to this list. I would like to print bar numbers that are "repeat
    aware",
    that is such that if bars 9 to 12 repeat twice, then the bar 
numbers go


      1, 2, …, 9, …, 12, 17, 18, etc.

    with a jump after 12 that corresponds to the bars 13 to 16 that are
    actually 9 to 12. Even better, I would like to print the two bar
    numbers
    for these bars (9/13, 10/14, etc.)


Perhaps this snippet could be of some use for you: 
http://lsr.di.unimi.it/LSR/Item?id=1080


I have been playing with this solution and I find it very satisfactory 
for my use. I do not understand yet all the intrinsics (in particular 
with respect to the context-spec-music and the make-apply-context 
functions), but I understand roughly how it works.


There is just one thing that I would want to "improve" (that is, for my 
use): the solution needs to map on all the music in order to find all 
the VoltaRepeatedMusic and override their callbacks to a newly defined 
one. This is achieved by providing a function "unfoldBarNumbers" that 
can be applied to any music and that does it.


I would however be interested to have that for all the music in my file, 
without having to call that function. The different ideas that pop into 
my mind in such a situation are the following:


1. Maybe I could override the "\repeat" function so that it does the 
same thing as by default but, when it is a "volta" repeat, it also 
attaches the new callback to the created music. I believe I see how to 
achieve that, except I seem unsuccessful in overriding the default 
definition of "\repeat". Even if I do something like:


#(define (repeat type num body alts)
  (display "\nrepeat\n"))

{ b4 \repeat volta 2 { b b } b }

it behaves as if I had not put the #(define ...) statement. I have 
played around this idea for a while without success. I have tried to see 
if "\repeat" had a special treatment (which it does, at least in the 
lexing/parsing), but I haven't been able to figure this out.


2. Maybe there is a way to define a function that will be ran before 
processing on any music? But this I have no idea if this is possible, or 
how to do it, and my research has proven ineffective so far.


3. Maybe there is a way to override the "make-volta-set" callback 
instead of creating a new one? Such that then, in LilyPond's normal 
execution, it uses the new definition and not and old one that then 
needs to be replaced?


If anyone has any idea on the topic, it would be really useful. In any 
case, I am quite interested in understanding how the execution works and 
thus where LilyPond finds the definition of "\repeat", since it is not 
where I was expecting it.


Thank you in advance, and thank you for your time!
— Niols

Re: Iterators in Scheme?

[Niols]

Hello Jean-Julien,

On 29/07/2020 17:28, Jean-Julien Fleck wrote:
Le mer. 29 juil. 2020 à 15:44, Niols > a écrit :



 > What do you want to achieve?

This is actually follow-up research from my side after a previous
e-mail
to this list. I would like to print bar numbers that are "repeat
aware",
that is such that if bars 9 to 12 repeat twice, then the bar numbers go

      1, 2, …, 9, …, 12, 17, 18, etc.

with a jump after 12 that corresponds to the bars 13 to 16 that are
actually 9 to 12. Even better, I would like to print the two bar
numbers
for these bars (9/13, 10/14, etc.)


Perhaps this snippet could be of some use for you: 
http://lsr.di.unimi.it/LSR/Item?id=1080


This is quite amazing!

This made me realise that my question is exactly the same as issue #5031 
[1]. The snippet you link is a workaround for this issue that does 
almost everything I am looking for. Also, it works in a way that is 
quite new to me, so understanding it will bring me quite a lot of 
interesting things; thank you!


[1] https://sourceforge.net/p/testlilyissues/issues/5031/

I will try to play with that and come back if I have more questions.

By default, I would also report back to the mailing-list, but I don't 
know if that is how it is done here. Do not hesitate to tell me if it is 
not the case on this mailing-list.


Best,
– Niols

Re: Iterators in Scheme?

[Jean-Julien Fleck]

Hello Niols,


Le mer. 29 juil. 2020 à 15:44, Niols  a écrit :

>
> > What do you want to achieve?
>
> This is actually follow-up research from my side after a previous e-mail
> to this list. I would like to print bar numbers that are "repeat aware",
> that is such that if bars 9 to 12 repeat twice, then the bar numbers go
>
>  1, 2, …, 9, …, 12, 17, 18, etc.
>
> with a jump after 12 that corresponds to the bars 13 to 16 that are
> actually 9 to 12. Even better, I would like to print the two bar numbers
> for these bars (9/13, 10/14, etc.)
>

Perhaps this snippet could be of some use for you:
http://lsr.di.unimi.it/LSR/Item?id=1080

Cheers,

-- 
JJ Fleck
Physique et Informatique
PCSI1 Lycée Kléber

Re: Iterators in Scheme?

[Niols]

Hi David,

On 29/07/2020 15:58, David Kastrup wrote:

Markup expressions are converted into stencils at a comparatively late
point of time.


So that means I could create the markups for the several bar numbers 
while I see them, and these markups would use a variable that contains 
the bar number of the end of the repeat, and by the time the markups are 
actually interpreted, I can have filled the variable with the right value?



 From what I understand, however, there are two passes: first the
iteration and then the engraving.


No, that's wrong.  It's just a single pass.  Iterators are strongly tied
to the sequence of music expressions in the source code (and only
persist while such an expression is being interpreted) while engravers
are tied to contexts and persist as long as the respective context
lives.  Iterators are driven by the music expressions and the advancing
time while engravers are driven by stream events announced in contexts.

But there is no separate pass involved.


Oh, OK, my bad. Thank you very much for the explanation!


For the problem of repeats, I read the code of the
Volta_repeat_iterator. In particular, it seems to read the
"repeat-count" property:

 rep_count_ = scm_to_int (get_property (get_music (), "repeat-count"));

but only uses it to detect the "first time" and the other times. The
first time:

 if (first_time_)
   {
 add_repeat_command (ly_symbol2scm ("start-repeat"));
 first_time_ = false;
   }

it adds a "start-repeat" command. As far as I understand, that means
that the engraving then only gets the repeat commands, which is
actually not enough information for what I am trying to achieve.

I am really motivated to read code and do my own research and write my
own Scheme code, but I do not really know where to look and if I
understood stuff correctly. If you had any pointer to understand more
(or better) about what I understood, and to achieve this in the end, I
would be very grateful.

In any case, I am already very grateful for the time you took to read
my message/s and answer me. Sorry for the long text.


I thought we had several options for bar numbering in connection with
repeats but don't know about the details.  There may be a more
specialised solution for your use case.


The only option of bar numbering with respect to repeat that I know of 
is with respect to the alternatives.


https://lilypond.org/doc/v2.18/Documentation/internals/bar_005fnumber_005fengraver

The Bar_number_engraver reads the alternativeNumberingStyle property 
that decides whether the numbers should keep increase, or be the same in 
all alternatives but with a letter, or just be the same in all alternatives.


If there is an other option, I have never found it in all the time I've 
crawled in the documentation.



Then there also is the recording-group-emulate function which you can
use for causing a complete pass of iteration just for event gathering
purposes.  It might be feasible to use that for collecting repeat count
information in advance.


So one can run the whole thing a first time, possibly with engravers 
gathering information, and then a second time with the acquired data. 
That sounds interesting. I am not sure about the repeat count 
information though.


Am I right to say the the music properties are accessible from iterators 
but not engravers? And thus that, unless an iterator puts the repeat 
count music property somewhere where an engraver can get it, I have no 
way of knowing how many times the repeat goes?


Thanks again for your time!

Best,
— Niols

Re: Iterators in Scheme?

[David Kastrup]

Niols  writes:

> Hi David,
>
> On 29/07/2020 15:16, David Kastrup wrote:
>> Niols  writes:
>>> Must I understand that it is not possible to write iterators in Scheme
>>> and that they have to be written in C++?
>> Yes.
>
> Thank you; at least I stop wondering about that.
>
>>> Must I then understand that, if I wanted to write iterators for a
>>> personal use, I would have to recompile LilyPond myself?
>> Yes, or reuse one of the existing ones.  Some are comparatively
>> generic.
>
> I would love that as it would allow to only have to write some
> Scheme. But I am not sure this can be achieved. See below.
>
>> What do you want to achieve?
>
> This is actually follow-up research from my side after a previous
> e-mail to this list. I would like to print bar numbers that are
> "repeat aware", that is such that if bars 9 to 12 repeat twice, then
> the bar numbers go
>
> 1, 2, …, 9, …, 12, 17, 18, etc.
>
> with a jump after 12 that corresponds to the bars 13 to 16 that are
> actually 9 to 12. Even better, I would like to print the two bar
> numbers for these bars (9/13, 10/14, etc.)
>
> I have tried hacking around the current bar number with an engraver
> that would recognize repeat commands, putting the current bar number
> in a stack whenever reaching a "start-repeat" and popping it when
> reaching an "end-repeat". This works, but:
>
> - does not seem very clean,
>
> - does not allow to write several bar numbers because we can't know
>   yet how long the repeat is going to be,

Markup expressions are converted into stencils at a comparatively late
point of time.

> - does not recognize repeats that are for more than twice.
>
> From what I understand, however, there are two passes: first the
> iteration and then the engraving.

No, that's wrong.  It's just a single pass.  Iterators are strongly tied
to the sequence of music expressions in the source code (and only
persist while such an expression is being interpreted) while engravers
are tied to contexts and persist as long as the respective context
lives.  Iterators are driven by the music expressions and the advancing
time while engravers are driven by stream events announced in contexts.

But there is no separate pass involved.

> For the problem of repeats, I read the code of the
> Volta_repeat_iterator. In particular, it seems to read the 
> "repeat-count" property:
>
> rep_count_ = scm_to_int (get_property (get_music (), "repeat-count"));
>
> but only uses it to detect the "first time" and the other times. The
> first time:
>
> if (first_time_)
>   {
> add_repeat_command (ly_symbol2scm ("start-repeat"));
> first_time_ = false;
>   }
>
> it adds a "start-repeat" command. As far as I understand, that means
> that the engraving then only gets the repeat commands, which is
> actually not enough information for what I am trying to achieve.
>
> I am really motivated to read code and do my own research and write my
> own Scheme code, but I do not really know where to look and if I 
> understood stuff correctly. If you had any pointer to understand more
> (or better) about what I understood, and to achieve this in the end, I 
> would be very grateful.
>
> In any case, I am already very grateful for the time you took to read
> my message/s and answer me. Sorry for the long text.

I thought we had several options for bar numbering in connection with
repeats but don't know about the details.  There may be a more
specialised solution for your use case.

Then there also is the recording-group-emulate function which you can
use for causing a complete pass of iteration just for event gathering
purposes.  It might be feasible to use that for collecting repeat count
information in advance.

-- 
David Kastrup

Re: Iterators in Scheme?

[Niols]

Hi David,

On 29/07/2020 15:16, David Kastrup wrote:

Niols  writes:

Must I understand that it is not possible to write iterators in Scheme
and that they have to be written in C++?


Yes.


Thank you; at least I stop wondering about that.


Must I then understand that, if I wanted to write iterators for a
personal use, I would have to recompile LilyPond myself?


Yes, or reuse one of the existing ones.  Some are comparatively generic.


I would love that as it would allow to only have to write some Scheme. 
But I am not sure this can be achieved. See below.



What do you want to achieve?


This is actually follow-up research from my side after a previous e-mail 
to this list. I would like to print bar numbers that are "repeat aware", 
that is such that if bars 9 to 12 repeat twice, then the bar numbers go


1, 2, …, 9, …, 12, 17, 18, etc.

with a jump after 12 that corresponds to the bars 13 to 16 that are 
actually 9 to 12. Even better, I would like to print the two bar numbers 
for these bars (9/13, 10/14, etc.)


I have tried hacking around the current bar number with an engraver that 
would recognize repeat commands, putting the current bar number in a 
stack whenever reaching a "start-repeat" and popping it when reaching an 
"end-repeat". This works, but:


- does not seem very clean,

- does not allow to write several bar numbers because we can't know yet 
how long the repeat is going to be,


- does not recognize repeats that are for more than twice.

From what I understand, however, there are two passes: first the 
iteration and then the engraving. Besides, the iteration gets access to 
all the information of the score, and thus also the number of repeats 
that is put by the user (N in "\repeat volta N"), while the engraving 
only get the events generated by the iteration.


For the problem of repeats, I read the code of the 
Volta_repeat_iterator. In particular, it seems to read the 
"repeat-count" property:


rep_count_ = scm_to_int (get_property (get_music (), "repeat-count"));

but only uses it to detect the "first time" and the other times. The 
first time:


if (first_time_)
  {
add_repeat_command (ly_symbol2scm ("start-repeat"));
first_time_ = false;
  }

it adds a "start-repeat" command. As far as I understand, that means 
that the engraving then only gets the repeat commands, which is actually 
not enough information for what I am trying to achieve.


I am really motivated to read code and do my own research and write my 
own Scheme code, but I do not really know where to look and if I 
understood stuff correctly. If you had any pointer to understand more 
(or better) about what I understood, and to achieve this in the end, I 
would be very grateful.


In any case, I am already very grateful for the time you took to read my 
message/s and answer me. Sorry for the long text.


Best,
— Niols

Re: Iterators in Scheme?

[David Kastrup]

Niols  writes:

> Hello,
>
> I am trying to dive a bit into the way LilyPond works, and I was
> wondering about iterators. The documentation contains a page named 
> “iterator tutorial” [1] but this is only a placeholder for a future
> tutorial. (I very much understand that it is painful to write such 
> tutorials and I am not here to complain of anything!)
>
> [1]
> https://lilypond.org/doc/v2.20/Documentation/contributor/iterator-tutorial
>
> The only thing the page actually says is “Iterators are routines
> written in C++ that process music expressions and sent the music
> events to the appropriate engravers and/or performers.”
>
> Must I understand that it is not possible to write iterators in Scheme
> and that they have to be written in C++?

Yes.

> Must I then understand that, if I wanted to write iterators for a
> personal use, I would have to recompile LilyPond myself?

Yes, or reuse one of the existing ones.  Some are comparatively generic.

> If it is actually possible to write an iterator in Scheme, would
> anyone have an example to show me or a link to give me?
>
> Thank you so much in advance.

What do you want to achieve?

-- 
David Kastrup

Iterators in Scheme?

[Niols]

Hello,

I am trying to dive a bit into the way LilyPond works, and I was 
wondering about iterators. The documentation contains a page named 
“iterator tutorial” [1] but this is only a placeholder for a future 
tutorial. (I very much understand that it is painful to write such 
tutorials and I am not here to complain of anything!)


[1] 
https://lilypond.org/doc/v2.20/Documentation/contributor/iterator-tutorial


The only thing the page actually says is “Iterators are routines written 
in C++ that process music expressions and sent the music events to the 
appropriate engravers and/or performers.”


Must I understand that it is not possible to write iterators in Scheme 
and that they have to be written in C++?


Must I then understand that, if I wanted to write iterators for a 
personal use, I would have to recompile LilyPond myself?


If it is actually possible to write an iterator in Scheme, would anyone 
have an example to show me or a link to give me?


Thank you so much in advance.

Best,
— Niols