Re: \fine, pre-process-in-final-translation-timestep & co.
On Jul 19, 2022, at 18:46, Jean Abou Samra wrote: > > Then we have the style of solution found in !1451, with per-engraver > heuristics. I honestly dislike this. I don't want to sound harsh, but "\fine does not end iteration" https://gitlab.com/lilypond/lilypond/-/merge_requests/1496 — Dan
Re: \fine, pre-process-in-final-translation-timestep & co.
Le 18/07/2022 à 22:21, Dan Eble a écrit : Imagining myself in the position of a user looking at a black box, I would not have much respect for a developer trying to make me believe that LilyPond needs more information than "\volta 2 \fine" to determine that the work ends there. I don't want to be that developer. I would give these examples: \version "2.23.11" \repeat segno 2 { \repeat volta 2 { c'1 1 1 1 1 1 1 1 } \volta 2 \fine \mark \default c'1 1 1 1 1 } vs. \repeat segno 2 { \repeat volta 2 { c'1 1 1 1 1 1 1 1 \tweak self-alignment-X #RIGHT \mark "Ad lib" } \volta 2 \fine c'1 1 1 1 1 } - \repeat segno 2 { \repeat volta 2 { c'1 1 1 1 1 1 1 } \volta 2 \fine \tempo "Lento" c'1 1 1 1 1 } vs. \repeat segno 2 { \repeat volta 2 { c'1 1 1 1 1 1 1 \tweak self-alignment-X #RIGHT \tweak font-size -3 \tempo "La seconda volta accelerando" } \volta 2 \fine c'1 1 1 1 1 } --- \repeat segno 2 { \repeat volta 2 { \once \override Staff.BarLine.color = red \bar ".|:" c'1 1 1 1 \mark \markup \with-color #red "Repeated twice" 1 1 1 \once \override Staff.BarLine.color = red } \volta 2 \fine c'1 1 1 1 1 } vs. \repeat segno 2 { c'1 1 1 1 1 1 1 \volta 2 { \tweak font-size -4 \mark \markup \with-color #red "Always use a double bar line at the end" \once \override Staff.BarLine.color = red \fine } c'1 1 1 1 1 } In an earlier message, you wrote: and "fully" explicit makes me chuckle in light of issue #34. I'm not sure what your reasoning is, but mine leads me to the opposite. The more I think about it, the more I prefer this option. The problem in issue #34 is exactly that the input does not specify how to order graces and events in other voices, so LilyPond is left to guess. As you can see from the examples above, we're not at risk of finding a perfect solution to that guessing game anytime soon. We are left with creating heuristics. At this point, I think I have exhausted my creativity for inventing such heuristics, without a satisfactory solution emerging that doesn't require a large rewrite of the code base that I won't be investing time into in the near future. Then we have the style of solution found in !1451, with per-engraver heuristics. I honestly dislike this. I don't want to sound harsh, but although a very nice feature, \fine is just one among hundreds of features in LilyPond. I'm not a fan of letting it spill all over the code base and requiring people who write and modify engravers (including users writing engravers in Scheme) to think about it. Is the benefit of slightly less verbose syntax really worth it? Not to mention that we probably need a way for the user to override the built-in heuristics if they fail. If you're really bothered by the redundancy, I think it should be possible to make this work: \repeat segno 2 { ... \alternative { % maybe, or maybe not require this \volta 1 { ... } } } Actually, this doesn't yield anything useful: \repeat volta 2 { c'1 1 1 1 \volta 2 \fine c'1 1 1 } whereas this does: \repeat volta 2 { c'1 1 1 1 \alternative { \volta 1 { c'1 1 1 } } } (and I wonder if \alternative should really be necessary, as discussed elsewhere). So this syntax looks more consistent to me, as it works similarly with \repeat volta and \repeat segno. That said, it's not that I really care all that much about the syntax. Above all, I would like to avoid adding logic to all/many unrelated engravers. PS: thinking again about this After all, it might be the only fully correct of doing it. Imagine a hairpin starting before fine and ending after. How steep the change in MIDI volume should be depends on its end moment and the absolute dynamic that is there, which can’t be known if translation is aborted at \fine. I have not found a meaningful case where it would occur.
Re: \fine, pre-process-in-final-translation-timestep & co.
On Jul 10, 2022, at 12:04, Jean Abou Samra wrote: > > The currently recommended syntax for DS al fine > repeats is > > \repeat segno 2 { > c'1 1 1 1 > \volta 2 \fine > c'1 1 1 1 > } > > This seems to work just as well, though: > > \repeat segno 2 { > c'1 1 1 1 > \volta 2 \fine > \volta 1 { c'1 1 1 1 } > } > > Do you see downsides with that syntax? Imagining myself in the position of a user looking at a black box, I would not have much respect for a developer trying to make me believe that LilyPond needs more information than "\volta 2 \fine" to determine that the work ends there. I don't want to be that developer. I won't rule it out, but it's not my first or second choice. — Dan
Re: \fine, pre-process-in-final-translation-timestep & co.
Le 09/07/2022 à 19:51, Jean Abou Samra a écrit : Repeats are complicated and I didn’t give this much thought, so it might make no sense, but you see the basic idea: \unfoldRepeats would be able to unfold the music in a way that doesn’t need interrupting translation with an event in the middle. In the above, which … an event at the point of fine was entered in determines whether it is included at the unfolded end. The currently recommended syntax for DS al fine repeats is \repeat segno 2 { c'1 1 1 1 \volta 2 \fine c'1 1 1 1 } This seems to work just as well, though: \repeat segno 2 { c'1 1 1 1 \volta 2 \fine \volta 1 { c'1 1 1 1 } } Do you see downsides with that syntax? If not, how about recommending it and letting \fine no longer abort translation outside of a folded repeat? The old syntax will be broken in \unfoldRepeats, but we could make \fine emit a warning if it appears before the normal end and outside of a folded repeat so that this won't go unnoticed. Jean
Re: \fine, pre-process-in-final-translation-timestep & co.
> Le 9 juil. 2022 à 22:08, Dan Eble a écrit : > > Do you have any concern about other engravers might have acknowledged the > grobs that are about to be killed, which might have performed interesting > actions assuming that the grobs would continue to exist on one system or > another? Does the battle-tested algorithm sort out all the consequences of > creating those grobs? I won’t know for sure until it has been tried out and all the code paths can be seen. I don’t anticipate major problems right now, although it has been a long day and I might be missing something. I see that I expressed myself poorly with ‘battle-tested’. It’s more like: the code is written to work that way and has received a lot of thought, so if break-visibility etc considerations are viewed as a sane way of doing the cut, that body of code can be profitably reused. For what it’s worth, doing it that way is _not_ a panacea in general either without some further thought: taken as-is, it will make clef/time/key changes right after fine print a dangling clef/time signature /key signature at the end. I’m not sure how to solve that in general. I think the idea of cutting spanners should work out well enough. For items, some more thinking is needed. Also, one thing you would need to be careful about is how references are rearranged (the equivalent of break substitution). While different systems live mostly independent lives after line breaking, some routines do need to look at other systems. I _guess_ the sanest way would be to eliminate those references completely instead of leaving them as references to dead grobs. In short: this is not the kind of idea I can crank out code for in two minutes.
Re: \fine, pre-process-in-final-translation-timestep & co.
Dan Eble writes: > On Jul 8, 2022, at 13:39, Jean Abou Samra wrote: >> >> Instead of rearranging the translation process to let \fine abort >> translation, let translation run normally until the end. >> >> Then, ‘cut’ the result. > ... >> For layout output, there is already a battle-tested algorithm, the >> one used to break into pieces after determining page breaking. I >> think it should be feasible to adapt it to run right after >> translation a first time to eliminate the part after \fine. More >> precisely, it should be as if a \break was present as \fine and the >> subsequent systems were not printed. > > Do you have any concern about other engravers might have acknowledged > the grobs that are about to be killed, which might have performed > interesting actions assuming that the grobs would continue to exist on > one system or another? Does the battle-tested algorithm sort out all > the consequences of creating those grobs? You mean, battle-tested like with \score { { \set Score.skipTypesetting = ##t \skip 1 } \midi { } } ? -- David Kastrup
Re: \fine, pre-process-in-final-translation-timestep & co.
On Jul 8, 2022, at 13:39, Jean Abou Samra wrote: > > Instead of rearranging the translation process to let \fine abort > translation, let translation run normally until the end. > > Then, ‘cut’ the result. ... > For layout output, there is already a battle-tested algorithm, the one used > to break into pieces after determining page breaking. I think it should be > feasible to adapt it to run right after translation a first time to eliminate > the part after \fine. More precisely, it should be as if a \break was present > as \fine and the subsequent systems were not printed. Do you have any concern about other engravers might have acknowledged the grobs that are about to be killed, which might have performed interesting actions assuming that the grobs would continue to exist on one system or another? Does the battle-tested algorithm sort out all the consequences of creating those grobs? — Dan
Re: \fine, pre-process-in-final-translation-timestep & co.
> Le 9 juil. 2022 à 18:28, Dan Eble a écrit : > > Are you serious enough about this suggestion to propose syntax that you would > consider acceptably explicit and that could actually be implemented? To be clear, I’m not trying to find syntax acceptably explicit for me; this is not a judgement. I just wonder if we can find syntax that is explicit, period, i.e. syntax that lets the user tell which events that should be kept at \fine and which that shouldn’t. My understanding is that there are two use cases for \fine right now: using it as a kind of memorable shorthand for \bar "|." at the end of the piece, and using it in \repeat segno to mark the end of the played music so that engraving can print "fine" and (unfoldRepeats-ed) MIDI can stop. The problem may be simpler if we accept to make separate commands for these two uses cases. For example, instead of \repeat segno 2 { … \volta 2 \fine … } one can imagine \repeat segno 2 { … \volta 1 { … } } Repeats are complicated and I didn’t give this much thought, so it might make no sense, but you see the basic idea: \unfoldRepeats would be able to unfold the music in a way that doesn’t need interrupting translation with an event in the middle. In the above, which … an event at the point of fine was entered in determines whether it is included at the unfolded end.
Re: \fine, pre-process-in-final-translation-timestep & co.
On Jul 9, 2022, at 11:56, Jean Abou Samra wrote: > > A third option: get out of this uncomfortable situation by > changing the syntax (it hasn't made it to a stable release > after all) so that which events are included and which > are not is fully explicit in the ly code. This idea is a bit vague to be called an option yet, and "fully" explicit makes me chuckle in light of issue #34. Are you serious enough about this suggestion to propose syntax that you would consider acceptably explicit and that could actually be implemented? I believe that I have done the best I can on it, so working on my own to find a better best would likely be a waste time. -- Dan
Re: \fine, pre-process-in-final-translation-timestep & co.
Le 09/07/2022 à 17:38, David Kastrup a écrit : Unmatched events are a problem in MIDI. MIDI processors and sequencers working with files may to some degree try to repair such problems, partly because they need to do so when cutting and pasting regions as well. But you don't get guarantees. Thanks for explaining this. I guess it would also be an option to emit end events at \fine but not start events, if they can be told apart. A third option: get out of this uncomfortable situation by changing the syntax (it hasn't made it to a stable release after all) so that which events are included and which are not is fully explicit in the ly code.
Re: \fine, pre-process-in-final-translation-timestep & co.
Jean Abou Samra writes: > Le 09/07/2022 à 16:31, Dan Eble a écrit : >> On Jul 8, 2022, at 13:39, Jean Abou Samra wrote: >>> Then, ‘cut’ the result. For MIDI, that shouldn’t be too hard: while >>> outputting, just stop where \fine was used. >> For the events that are simultaneous with the \fine, would you emit them or >> not? >> If you would emit them, then you would emit note-on events. >> If you would not emit them, then you would not emit note-off events. > > Is it a problem not to emit note-off events at the end of the piece if > it's going to end anyway? Unmatched events are a problem in MIDI. MIDI processors and sequencers working with files may to some degree try to repair such problems, partly because they need to do so when cutting and pasting regions as well. But you don't get guarantees. -- David Kastrup
Re: \fine, pre-process-in-final-translation-timestep & co.
Le 09/07/2022 à 16:22, Dan Eble a écrit : On Jul 8, 2022, at 13:39, Jean Abou Samra wrote: For example, whether an item on the non-musical column at the point of \fine is included depends on whether its break-visibility makes it visible at EOL. The fine text itself will thus be included, while a rehearsal mark won’t. That is just as expected. Not exactly. ``` void Mark_engraver::finalize () { if (final_text_) { // A mark created at the very end is always visible even if it would not // be visible at the end of a broken line. set_property (final_text_, "break-visibility", scm_c_make_vector (3, SCM_BOOL_T)); } final_text_ = nullptr; } ``` Perhaps. On the other hand, would we really want that behavior for a premature end at \fine? The most likely case is a \mark \default marking the start of the section after \fine, which shouldn't be displayed if that section isn't printed.
Re: \fine, pre-process-in-final-translation-timestep & co.
On Jul 9, 2022, at 10:38, Jean Abou Samra wrote: > > Le 09/07/2022 à 16:31, Dan Eble a écrit : >> On Jul 8, 2022, at 13:39, Jean Abou Samra wrote: >>> Then, ‘cut’ the result. For MIDI, that shouldn’t be too hard: while >>> outputting, just stop where \fine was used. >> For the events that are simultaneous with the \fine, would you emit them or >> not? >> If you would emit them, then you would emit note-on events. >> If you would not emit them, then you would not emit note-off events. > > Is it a problem not to emit note-off events at the end of the piece if it's > going to end anyway? Good point. I assumed it would be, but I would have to research that. — Dan
Re: \fine, pre-process-in-final-translation-timestep & co.
Le 09/07/2022 à 16:31, Dan Eble a écrit : On Jul 8, 2022, at 13:39, Jean Abou Samra wrote: Then, ‘cut’ the result. For MIDI, that shouldn’t be too hard: while outputting, just stop where \fine was used. For the events that are simultaneous with the \fine, would you emit them or not? If you would emit them, then you would emit note-on events. If you would not emit them, then you would not emit note-off events. Is it a problem not to emit note-off events at the end of the piece if it's going to end anyway?
Re: \fine, pre-process-in-final-translation-timestep & co.
On Jul 8, 2022, at 13:39, Jean Abou Samra wrote: > > Then, ‘cut’ the result. For MIDI, that shouldn’t be too hard: while > outputting, just stop where \fine was used. For the events that are simultaneous with the \fine, would you emit them or not? If you would emit them, then you would emit note-on events. If you would not emit them, then you would not emit note-off events. — Dan
Re: \fine, pre-process-in-final-translation-timestep & co.
On Jul 8, 2022, at 13:39, Jean Abou Samra wrote: > For example, whether an item on the non-musical column at the point of \fine > is included depends on whether its break-visibility makes it visible at EOL. > The fine text itself will thus be included, while a rehearsal mark won’t. > That is just as expected. Not exactly. ``` void Mark_engraver::finalize () { if (final_text_) { // A mark created at the very end is always visible even if it would not // be visible at the end of a broken line. set_property (final_text_, "break-visibility", scm_c_make_vector (3, SCM_BOOL_T)); } final_text_ = nullptr; } ``` — Dan
Re: \fine, pre-process-in-final-translation-timestep & co.
> Le 5 juil. 2022 à 08:11, Jean Abou Samra a écrit : > > Neither am I, but it is the only idea I could propose. Well, here is another, completely different one. Instead of rearranging the translation process to let \fine abort translation, let translation run normally until the end. Then, ‘cut’ the result. For MIDI, that shouldn’t be too hard: while outputting, just stop where \fine was used. For layout output, there is already a battle-tested algorithm, the one used to break into pieces after determining page breaking. I think it should be feasible to adapt it to run right after translation a first time to eliminate the part after \fine. More precisely, it should be as if a \break was present as \fine and the subsequent systems were not printed. For example, whether an item on the non-musical column at the point of \fine is included depends on whether its break-visibility makes it visible at EOL. The fine text itself will thus be included, while a rehearsal mark won’t. That is just as expected. After all, it might be the only fully correct of doing it. Imagine a hairpin starting before fine and ending after. How steep the change in MIDI volume should be depends on its end moment and the absolute dynamic that is there, which can’t be known if translation is aborted at \fine. Jean
Re: \fine, pre-process-in-final-translation-timestep & co.
On 7/5/22 18:39, Dan Eble wrote: I'll try to explain this more clearly. I doubt that we actually disagree about this. I did not mean that those two cases should be handled the same in every respect. My thoughts were focused on creating spanners at the end of the score. Warning about a score ending in c1*0\< is likely to be helpful. Creating a weird hairpin grob because no \fine was observed is not likely to be helpful; rather, it is likely to lead to other errors like in issue 6372. This I agree with. It does mean we need to differentiate termination with \fine and normal termination, which I think we also agree on.
Re: \fine, pre-process-in-final-translation-timestep & co.
> On Jul 5, 2022, at 08:05, Dan Eble wrote: > > On Jul 5, 2022, at 02:11, Jean Abou Samra wrote: >> >> On 7/5/22 02:03, Dan Eble wrote: >>> Don't focus too closely on \fine. Engraving in the final timestep should >>> be orderly whether it is caused by \fine or the natural end of the input. >>> You're just more likely to get into interesting situations by something >>> sane like >>> >>>… \fine c1\< … >>> >>> than by something crazy like >>> >>>… c1*0\< >> >> >> >> I actually disagree. For me, an ideal design engraves >> { ... \fine c1\< } just fine, but warns upon seeing >> { ... c1*0\< } because that sounds like a mistake and >> a diagnostic is helpful. > > I didn't say there should be no warning. I said engraving should be orderly. > Do we agree that c1*0\< should not warn AND THEN create an unusual spanner > anyway, risking downstream errors like issue 6372 [1]? I'm pretty sure we do. I'll try to explain this more clearly. I doubt that we actually disagree about this. I did not mean that those two cases should be handled the same in every respect. My thoughts were focused on creating spanners at the end of the score. Warning about a score ending in c1*0\< is likely to be helpful. Creating a weird hairpin grob because no \fine was observed is not likely to be helpful; rather, it is likely to lead to other errors like in issue 6372. -- Dan
Re: \fine, pre-process-in-final-translation-timestep & co.
On Jul 5, 2022, at 02:11, Jean Abou Samra wrote: > > On 7/5/22 02:03, Dan Eble wrote: >> Don't focus too closely on \fine. Engraving in the final timestep should be >> orderly whether it is caused by \fine or the natural end of the input. >> You're just more likely to get into interesting situations by something sane >> like >> >> … \fine c1\< … >> >> than by something crazy like >> >> … c1*0\< > > > > I actually disagree. For me, an ideal design engraves > { ... \fine c1\< } just fine, but warns upon seeing > { ... c1*0\< } because that sounds like a mistake and > a diagnostic is helpful. I didn't say there should be no warning. I said engraving should be orderly. Do we agree that c1*0\< should not warn AND THEN create an unusual spanner anyway, risking downstream errors like issue 6372 [1]? I'm pretty sure we do. — Dan [1] https://gitlab.com/lilypond/lilypond/-/issues/6372
Re: \fine, pre-process-in-final-translation-timestep & co.
On 7/5/22 02:03, Dan Eble wrote: Do you envision having the framework record events as they are dispatched to pre-listeners and then replay them to dispatch them to post-listeners? Yes. Would you expect events to be dispatched to post-listeners in the same global order as the pre-listeners? in no specified order? something in between? The order of listeners is already unspecified, so that shouldn't matter much, although it would probably be easier to think about if the order were the same. Would you allow an engraver to register both a pre-listener and post-listener method for the same event type? Yes. For example, it would be possible to have a pre-listener to set forbidBreak and a post-listener to create a grob. How would you expect something like Dynamic_engraver to work? It currently listens for span_dynamic events, but acts differently depending on the event's span-direction property. From the rest of your message, I gather that you would want it to hear the end event with a pre-listener and the start event with a post-listener. ``` void Dynamic_engraver::listen_span_dynamic (Stream_event *ev) { Direction d = from_scm (get_property (ev, "span-direction")); assign_event_once (accepted_spanevents_drul_[d], ev); } ``` Interesting, that gives me food for thought. I have actually long been thinking we ought to have separate listeners for the start and the end event in the case of span events, like we have end acknowledgers for spanners. Without that, the easiest would be (pre-listeners ((span-dynamic-event engraver event) (when (eqv? LEFT (ly:event-property event 'span-direction)) ...))) (post-listeners ((span-dynamic-event engraver event) (when (eqv? RIGHT (ly:event-property event 'span-direction)) ...))) Same for acknowledgers, lots of accumulating in vectors and clearing those vectors can be eliminated. I'm not contradicting this, but I would want to verify it before designing around it. A quick scan suggests Accidental_engraver Axis_group_engraver Dynamic_align_engraver Fingering_column_engraver New_fingering_engraver Trill_spanner_engraver My understanding so far is that you need to exclude some events in engravers because \fine, like overrides, can come at any time during the "listeners" phase. If an engraver has already recorded the event in its listener, it needs to "unrecord" it somehow. There are many engravers that work this way. There are also engravers that create grobs when they "acknowledge" other grobs (e.g. Hyphen_engraver). There are also engravers that create grobs when a context property is set (e.g. Mark_engraver). Engraver of the former type don't need changes in connection with \fine, do they? I guess Mark_engraver would be fixed by using post-listeners in Mark_tracking_translator. Well, for wha it's worth, the expected behavior is not clear. { c'1 \fine \mark \default %% Not expected to be printed c'1 } vs. { c'1 \mark \markup \musicglyph "scripts.ufermata" %% Expected \fine c'1 } (which emphasizes the need for something else than \mark for markups once again ...) That could become very natural in the model above: if you saw \fine (doing that in a pre-listener), just don't run any post-listeners. Don't focus too closely on \fine. Engraving in the final timestep should be orderly whether it is caused by \fine or the natural end of the input. You're just more likely to get into interesting situations by something sane like … \fine c1\< … than by something crazy like … c1*0\< I actually disagree. For me, an ideal design engraves { ... \fine c1\< } just fine, but warns upon seeing { ... c1*0\< } because that sounds like a mistake and a diagnostic is helpful. Jump straight to post-process-music. post-listeners and post-process-music are assumed to contain everything that creates grobs from events. Where should a hypothetical pedagogical spanner over the final barline be started? I guess it would be in post-process-music by necessity. Ah, you caught me there. I have to think about this one some more. Right now, I don't have anything better than a pre-listener recording the event like in the current model. When post-process-music creates grobs from events, those events are assumed to have been recorded in post-listeners. I'm not sure what conclusions to draw from this. Neither am I, but it is the only idea I could propose. Jean
Re: \fine, pre-process-in-final-translation-timestep & co.
On Jul 4, 2022, at 17:12, Jean Abou Samra wrote: > > | | pre-listeners <- RENAMING > | | > | pre-process-music => | pre-process-music > | | > | | post-listeners <- NEW STEP Do you envision having the framework record events as they are dispatched to pre-listeners and then replay them to dispatch them to post-listeners? Would you expect events to be dispatched to post-listeners in the same global order as the pre-listeners? in no specified order? something in between? Would you allow an engraver to register both a pre-listener and post-listener method for the same event type? How would you expect something like Dynamic_engraver to work? It currently listens for span_dynamic events, but acts differently depending on the event's span-direction property. From the rest of your message, I gather that you would want it to hear the end event with a pre-listener and the start event with a post-listener. ``` void Dynamic_engraver::listen_span_dynamic (Stream_event *ev) { Direction d = from_scm (get_property (ev, "span-direction")); assign_event_once (accepted_spanevents_drul_[d], ev); } ``` > Same for acknowledgers, lots of accumulating in vectors and clearing > those vectors can be eliminated. I'm not contradicting this, but I would want to verify it before designing around it. > My understanding so far is that you need to exclude some events > in engravers because \fine, like overrides, can come at > any time during the "listeners" phase. If an engraver has > already recorded the event in its listener, it needs to > "unrecord" it somehow. There are many engravers that work this way. There are also engravers that create grobs when they "acknowledge" other grobs (e.g. Hyphen_engraver). There are also engravers that create grobs when a context property is set (e.g. Mark_engraver). > That could become very natural in the model above: if you > saw \fine (doing that in a pre-listener), just don't run > any post-listeners. Don't focus too closely on \fine. Engraving in the final timestep should be orderly whether it is caused by \fine or the natural end of the input. You're just more likely to get into interesting situations by something sane like … \fine c1\< … than by something crazy like … c1*0\< > Jump straight to post-process-music. > post-listeners and post-process-music are assumed to contain > everything that creates grobs from events. Where should a hypothetical pedagogical spanner over the final barline be started? I guess it would be in post-process-music by necessity. > When > post-process-music creates grobs from events, those events > are assumed to have been recorded in post-listeners. I'm not sure what conclusions to draw from this. — Dan