Re: Syntax propertize in font-latex (Re: bug#35140: 12.1; Scan errors in doctex mode with ^^A-comments after braces)
> Stefan Monnier writes: > `syntax-propertize-via-font-lock' is not even declared obsolete (yet?), > so yes, it will survive. I see, thanks! Regards, Ikumi Keita #StandWithUkraine #StopWarInUkraine
Re: Syntax propertize in font-latex (Re: bug#35140: 12.1; Scan errors in doctex mode with ^^A-comments after braces)
> I see `syntax-propertize-via-font-lock' is implemented using > `font-lock-syntactic-keywords' as seen below. Is it planned that > `syntax-propertize-via-font-lock' survives after > `font-lock-syntactic-keywords' is removed in Emacs 29? `syntax-propertize-via-font-lock' is not even declared obsolete (yet?), so yes, it will survive. Stefan
Re: Syntax propertize in font-latex (Re: bug#35140: 12.1; Scan errors in doctex mode with ^^A-comments after braces)
Hi Tassilo and Stefan, > Tassilo Horn writes: > Ikumi Keita writes: >> The problems is that it requires some changes in entries in >> `font-latex-syntactic-keywords', sometimes rather non-trivial ones as >> seen in the patch. It's possible to do required changes in files >> distributed with AUCTeX, but if some user adds entries to >> `font-latex-syntactic-keywords' in their private styles, this >> incompatibility can be a non-easy problem. > Indeed, and therefore I'd like to avoid such a drastic change if > possible. I see `syntax-propertize-via-font-lock' is implemented using `font-lock-syntactic-keywords' as seen below. Is it planned that `syntax-propertize-via-font-lock' survives after `font-lock-syntactic-keywords' is removed in Emacs 29? (If not, we have to consider to migrate to method using `syntax-propertize-rules', introducing breaking changes in the format of `font-latex-syntactic-keywords'.) Current implementation of `syntax-propertize-via-font-lock': -- (defun syntax-propertize-via-font-lock (keywords) "Propertize for syntax using font-lock syntax. KEYWORDS obeys the format used in `font-lock-syntactic-keywords'. The return value is a function (with two parameters, START and END) suitable for `syntax-propertize-function'." (lambda (start end) (with-no-warnings (let ((font-lock-syntactic-keywords keywords)) (font-lock-fontify-syntactic-keywords-region start end) ;; In case it was eval'd/compiled. (setq keywords font-lock-syntactic-keywords) -- Regards, Ikumi Keita #StandWithUkraine #StopWarInUkraine
Re: Syntax propertize in font-latex (Re: bug#35140: 12.1; Scan errors in doctex mode with ^^A-comments after braces)
Ikumi Keita writes: >>> BTW, I noticed in the patch that `font-latex.el` appears to still >>> make use of `font-lock-syntactic-keywords`. This var has been >>> obsolete since Emacs-24.1 and it's like to disappear in Emacs-29, so >>> it would be good to fix this issue (maybe using >>> `syntax-propertize-via-font-lock`). > >> Thanks for the heads-up, I've done that just now. > >> >> https://git.savannah.gnu.org/cgit/auctex.git/commit/?id=ab3bfaf1033f0d6e0ecfe3172794a6363450b207 > > Unfortunately, this breaks regression test and results in: > SUMMARY OF TEST RESULTS > --- > Files examined: 13 > Ran 49 tests, 35 results as expected, 11 unexpected, 3 skipped > 3 files contained unexpected results: > latex/latex-test.log > latex/font-latex-test.log > context/context-test.log Oh, sorry. :-) > I encountered the same problem when I tried similar approach before. The > reasons of these errors is that > 1. `indent-according-to-mode' requires that syntax propertization is >done. > 2. `syntax-propertize-via-font-lock' needs font lock has been >initialized in that buffer. > 3. In batch mode, tests run before font lock is initialized because it >doesn't enter command wait loop. For example, > (with-temp-buffer > (LaTeX-mode) > syntax-propertize-function) >returns nil. I've just tried adding a (font-lock-ensure) in some tests which fixes them. > Attached is my preliminary attempt to avoid the errors by > circumventing `syntax-propertize-via-font-lock'. (Sorry, it involves > some unrelated changes.) Though I haven't finished it yet, I expect it > works basically and doesn't interfere with regression tests. I'm not sure I've actually separated the unrelated changes from the fix for the issue at hand but it looks much more complex, doesn't it? IMO, that the tests fail is just an artifact of the tests running in batch mode which will never happen in any real editing context. So I wouldn't want to jump though hoops to fix the tests. Maybe the simplest fix could be to add an advice (define-advice LaTeX-mode (:after ()) (font-lock-ensure)) so that our tests can be sure font-lock has been done after LaTeX-mode has been activated. > The problems is that it requires some changes in entries in > `font-latex-syntactic-keywords', sometimes rather non-trivial ones as > seen in the patch. It's possible to do required changes in files > distributed with AUCTeX, but if some user adds entries to > `font-latex-syntactic-keywords' in their private styles, this > incompatibility can be a non-easy problem. Indeed, and therefore I'd like to avoid such a drastic change if possible. FWIW, the following patch fixes all tests. Bye, Tassilo --8<---cut here---start->8--- 3 files changed, 9 insertions(+) tests/context/context-test.el | 3 +++ tests/latex/font-latex-test.el | 3 +++ tests/latex/latex-test.el | 3 +++ modified tests/context/context-test.el @@ -24,6 +24,9 @@ (require 'ert) (require 'context) +(define-advice ConTeXt-mode-common-initialization (:after ()) + (font-lock-ensure)) + (AUCTeX-set-ert-path 'ConTeXt-indent-test/in "context-indentation-in.tex" modified tests/latex/font-latex-test.el @@ -27,6 +27,9 @@ (defvar font-lock-beg) (defvar font-lock-end) +(define-advice LaTeX-common-initialization (:after ()) + (font-lock-ensure)) + (ert-deftest font-latex-three-dollars () "Test three consecutive dollar is ignored." ;; When the function `font-latex-match-dollar-math' encounters three modified tests/latex/latex-test.el @@ -24,6 +24,9 @@ (require 'ert) (require 'latex) +(define-advice LaTeX-common-initialization (:after ()) + (font-lock-ensure)) + (AUCTeX-set-ert-path 'LaTeX-indent-tabular-test/in "tabular-in.tex" --8<---cut here---end--->8---
Syntax propertize in font-latex (Re: bug#35140: 12.1; Scan errors in doctex mode with ^^A-comments after braces)
[ Changed Cc: to auctex-devel. ]
> Tassilo Horn writes:
> Stefan Monnier via bug-auctex via Bug reporting list for AUCTeX
> writes:
>> BTW, I noticed in the patch that `font-latex.el` appears to still make
>> use of `font-lock-syntactic-keywords`. This var has been obsolete
>> since Emacs-24.1 and it's like to disappear in Emacs-29, so it would
>> be good to fix this issue (maybe using
>> `syntax-propertize-via-font-lock`).
> Thanks for the heads-up, I've done that just now.
>
> https://git.savannah.gnu.org/cgit/auctex.git/commit/?id=ab3bfaf1033f0d6e0ecfe3172794a6363450b207
Unfortunately, this breaks regression test and results in:
SUMMARY OF TEST RESULTS
---
Files examined: 13
Ran 49 tests, 35 results as expected, 11 unexpected, 3 skipped
3 files contained unexpected results:
latex/latex-test.log
latex/font-latex-test.log
context/context-test.log
I encountered the same problem when I tried similar approach before. The
reasons of these errors is that
1. `indent-according-to-mode' requires that syntax propertization is
done.
2. `syntax-propertize-via-font-lock' needs font lock has been
initialized in that buffer.
3. In batch mode, tests run before font lock is initialized because it
doesn't enter command wait loop. For example,
(with-temp-buffer
(LaTeX-mode)
syntax-propertize-function)
returns nil.
Attached is my preliminary attempt to avoid the errors by circumventing
`syntax-propertize-via-font-lock'. (Sorry, it involves some unrelated
changes.) Though I haven't finished it yet, I expect it works basically
and doesn't interfere with regression tests.
The problems is that it requires some changes in entries in
`font-latex-syntactic-keywords', sometimes rather non-trivial ones as
seen in the patch. It's possible to do required changes in files
distributed with AUCTeX, but if some user adds entries to
`font-latex-syntactic-keywords' in their private styles, this
incompatibility can be a non-easy problem.
Regards,
Ikumi Keita
#StandWithUkraine #StopWarInUkraine
diff --git a/font-latex.el b/font-latex.el
index 8b1fec60..1468edd6 100644
--- a/font-latex.el
+++ b/font-latex.el
@@ -1003,9 +1003,6 @@ The car is used for subscript, the cdr is used for superscripts."
The form should be the same as in `font-lock-syntactic-keywords'.")
(make-variable-buffer-local 'font-latex-syntactic-keywords-extra)
-;; Set and updated in `font-latex-set-syntactic-keywords'.
-(defvar font-latex-doctex-syntactic-keywords nil)
-
(defun font-latex-set-syntactic-keywords ()
"Set the variable `font-latex-syntactic-keywords'.
This function can be used to refresh the variable in case other
@@ -1028,8 +1025,7 @@ have changed."
(regexp-opt verb-macros-with-braces))
font-latex-syntactic-keywords nil)
(unless (= (length verb-envs) 0)
- (add-to-list
- 'font-latex-syntactic-keywords
+ (push
`(,(concat
"^[ \t]*begin *{\\(?:" verb-envs "\\)}"
;; Some environments accept an optional and/or mandatory
@@ -1053,9 +1049,17 @@ have changed."
;; catches the case where verbatim content is written
;; immediately after the \begin{verbatim}.
"\\(\n\\|.\\)")
- (1 "|" t)))
- (add-to-list
- 'font-latex-syntactic-keywords
+ (1 "|")
+ ;; Span non-nil `syntax-multiline' property over
+ ;; \begin{verbatim}[WHOLE OPTIONAL ARGUMENT] so that editing
+ ;; inside the multi-line optional arguments won't turn off
+ ;; verbatim faces inside the environments even when those
+ ;; contents are very large.
+ (0 (ignore
+ (put-text-property (match-beginning 0) (match-end 0)
+'syntax-multiline t
+ font-latex-syntactic-keywords)
+ (push
;; Using the newline character for the syntax property often
;; resulted in fontification problems when text was inserted at
;; the end of the verbatim environment. That's why we now use
@@ -1063,10 +1067,10 @@ have changed."
;; `font-latex-make-user-keywords' to remove the spurious
;; fontification of the backslash.
`(,(concat "\\(\\)end *{\\(?:" verb-envs "\\)}")
- (1 "|" t
+ (1 "|"))
+ font-latex-syntactic-keywords))
(unless (= (length verb-macros-with-delims) 0)
- (add-to-list
- 'font-latex-syntactic-keywords
+ (push
`(,(concat "\\(?:" verb-macros-with-delims "\\)"
;; Some macros take an optional argument. This is
;; the same line as above for environments.
@@ -1074,29 +1078,46 @@ have changed."
;; An opening curly brace as delimiter is valid, but
;; allowing it might screw up fontification of stuff
;; like "\url{...} foo \textbf{<--!...}".
- "\\([^a-z@*\n\f{]\\).*?"
+
