branch: externals/wisi commit d15913f4fd6a5bc545c2a656645b44fc123c4979 Author: Stephen Leake <stephen_le...@stephe-leake.org> Commit: Stephen Leake <stephen_le...@stephe-leake.org>
Release version 3.1.5 * .gitignore: More build artifacts. * NEWS: Fix release date. * doclicense.texi: New file. * wisi.info: Delete; elpa builds it, so no longer in git. * README: Bump version. * wisi.el: * wisi.texi: * wisitoken-user_guide.texinfo: New file. --- .gitignore | 3 + NEWS | 2 +- README | 2 +- doclicense.texi | 507 +++++++++++++++++ wisi.el | 4 +- wisi.info | 1294 ------------------------------------------ wisi.texi | 4 +- wisitoken-user_guide.info | 611 -------------------- wisitoken-user_guide.texinfo | 616 ++++++++++++++++++++ 9 files changed, 1132 insertions(+), 1911 deletions(-) diff --git a/.gitignore b/.gitignore index 5f08039..cb87271 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,6 @@ /wisi-pkg.el /wisi-autoloads.el *.elc +obj/ +wisi.gpr +wisitoken-user_guide.info diff --git a/NEWS b/NEWS index 952e6c7..9264d77 100644 --- a/NEWS +++ b/NEWS @@ -7,7 +7,7 @@ Please send wisi bug reports to bug-gnu-em...@gnu.org, with * wisi 3.1.5 -29 Jul 2021 +30 Jul 2021 ** Update several SAL files for compatibility with gnat FSF 11, Pro 22, Community 2021. diff --git a/README b/README index 7e3ef01..6132ef2 100644 --- a/README +++ b/README @@ -1,4 +1,4 @@ -Emacs wisi package 3.1.3 +Emacs wisi package 3.1.5 The wisi package provides utilities for using generalized error-correcting LR parsers (in external processes) to do indentation, diff --git a/doclicense.texi b/doclicense.texi new file mode 100644 index 0000000..a511ffc --- /dev/null +++ b/doclicense.texi @@ -0,0 +1,507 @@ +@c -*-texinfo-*- +@c The GNU Free Documentation License. +@center Version 1.3, 3 November 2008 + +@c This file is intended to be included within another document, +@c hence no sectioning command or @node. + +@display +Copyright @copyright{} 2000, 2001, 2002, 2007, 2008, 2009 Free Software Foundation, Inc. +@uref{http://fsf.org/} + +Everyone is permitted to copy and distribute verbatim copies +of this license document, but changing it is not allowed. +@end display + +@enumerate 0 +@item +PREAMBLE + +The purpose of this License is to make a manual, textbook, or other +functional and useful document @dfn{free} in the sense of freedom: to +assure everyone the effective freedom to copy and redistribute it, +with or without modifying it, either commercially or noncommercially. +Secondarily, this License preserves for the author and publisher a way +to get credit for their work, while not being considered responsible +for modifications made by others. + +This License is a kind of ``copyleft'', which means that derivative +works of the document must themselves be free in the same sense. It +complements the GNU General Public License, which is a copyleft +license designed for free software. + +We have designed this License in order to use it for manuals for free +software, because free software needs free documentation: a free +program should come with manuals providing the same freedoms that the +software does. But this License is not limited to software manuals; +it can be used for any textual work, regardless of subject matter or +whether it is published as a printed book. We recommend this License +principally for works whose purpose is instruction or reference. + +@item +APPLICABILITY AND DEFINITIONS + +This License applies to any manual or other work, in any medium, that +contains a notice placed by the copyright holder saying it can be +distributed under the terms of this License. Such a notice grants a +world-wide, royalty-free license, unlimited in duration, to use that +work under the conditions stated herein. The ``Document'', below, +refers to any such manual or work. Any member of the public is a +licensee, and is addressed as ``you''. You accept the license if you +copy, modify or distribute the work in a way requiring permission +under copyright law. + +A ``Modified Version'' of the Document means any work containing the +Document or a portion of it, either copied verbatim, or with +modifications and/or translated into another language. + +A ``Secondary Section'' is a named appendix or a front-matter section +of the Document that deals exclusively with the relationship of the +publishers or authors of the Document to the Document's overall +subject (or to related matters) and contains nothing that could fall +directly within that overall subject. (Thus, if the Document is in +part a textbook of mathematics, a Secondary Section may not explain +any mathematics.) The relationship could be a matter of historical +connection with the subject or with related matters, or of legal, +commercial, philosophical, ethical or political position regarding +them. + +The ``Invariant Sections'' are certain Secondary Sections whose titles +are designated, as being those of Invariant Sections, in the notice +that says that the Document is released under this License. If a +section does not fit the above definition of Secondary then it is not +allowed to be designated as Invariant. The Document may contain zero +Invariant Sections. If the Document does not identify any Invariant +Sections then there are none. + +The ``Cover Texts'' are certain short passages of text that are listed, +as Front-Cover Texts or Back-Cover Texts, in the notice that says that +the Document is released under this License. A Front-Cover Text may +be at most 5 words, and a Back-Cover Text may be at most 25 words. + +A ``Transparent'' copy of the Document means a machine-readable copy, +represented in a format whose specification is available to the +general public, that is suitable for revising the document +straightforwardly with generic text editors or (for images composed of +pixels) generic paint programs or (for drawings) some widely available +drawing editor, and that is suitable for input to text formatters or +for automatic translation to a variety of formats suitable for input +to text formatters. A copy made in an otherwise Transparent file +format whose markup, or absence of markup, has been arranged to thwart +or discourage subsequent modification by readers is not Transparent. +An image format is not Transparent if used for any substantial amount +of text. A copy that is not ``Transparent'' is called ``Opaque''. + +Examples of suitable formats for Transparent copies include plain +@sc{ascii} without markup, Texinfo input format, La@TeX{} input +format, @acronym{SGML} or @acronym{XML} using a publicly available +@acronym{DTD}, and standard-conforming simple @acronym{HTML}, +PostScript or @acronym{PDF} designed for human modification. Examples +of transparent image formats include @acronym{PNG}, @acronym{XCF} and +@acronym{JPG}. Opaque formats include proprietary formats that can be +read and edited only by proprietary word processors, @acronym{SGML} or +@acronym{XML} for which the @acronym{DTD} and/or processing tools are +not generally available, and the machine-generated @acronym{HTML}, +PostScript or @acronym{PDF} produced by some word processors for +output purposes only. + +The ``Title Page'' means, for a printed book, the title page itself, +plus such following pages as are needed to hold, legibly, the material +this License requires to appear in the title page. For works in +formats which do not have any title page as such, ``Title Page'' means +the text near the most prominent appearance of the work's title, +preceding the beginning of the body of the text. + +The ``publisher'' means any person or entity that distributes copies +of the Document to the public. + +A section ``Entitled XYZ'' means a named subunit of the Document whose +title either is precisely XYZ or contains XYZ in parentheses following +text that translates XYZ in another language. (Here XYZ stands for a +specific section name mentioned below, such as ``Acknowledgements'', +``Dedications'', ``Endorsements'', or ``History''.) To ``Preserve the Title'' +of such a section when you modify the Document means that it remains a +section ``Entitled XYZ'' according to this definition. + +The Document may include Warranty Disclaimers next to the notice which +states that this License applies to the Document. These Warranty +Disclaimers are considered to be included by reference in this +License, but only as regards disclaiming warranties: any other +implication that these Warranty Disclaimers may have is void and has +no effect on the meaning of this License. + +@item +VERBATIM COPYING + +You may copy and distribute the Document in any medium, either +commercially or noncommercially, provided that this License, the +copyright notices, and the license notice saying this License applies +to the Document are reproduced in all copies, and that you add no other +conditions whatsoever to those of this License. You may not use +technical measures to obstruct or control the reading or further +copying of the copies you make or distribute. However, you may accept +compensation in exchange for copies. If you distribute a large enough +number of copies you must also follow the conditions in section 3. + +You may also lend copies, under the same conditions stated above, and +you may publicly display copies. + +@item +COPYING IN QUANTITY + +If you publish printed copies (or copies in media that commonly have +printed covers) of the Document, numbering more than 100, and the +Document's license notice requires Cover Texts, you must enclose the +copies in covers that carry, clearly and legibly, all these Cover +Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on +the back cover. Both covers must also clearly and legibly identify +you as the publisher of these copies. The front cover must present +the full title with all words of the title equally prominent and +visible. You may add other material on the covers in addition. +Copying with changes limited to the covers, as long as they preserve +the title of the Document and satisfy these conditions, can be treated +as verbatim copying in other respects. + +If the required texts for either cover are too voluminous to fit +legibly, you should put the first ones listed (as many as fit +reasonably) on the actual cover, and continue the rest onto adjacent +pages. + +If you publish or distribute Opaque copies of the Document numbering +more than 100, you must either include a machine-readable Transparent +copy along with each Opaque copy, or state in or with each Opaque copy +a computer-network location from which the general network-using +public has access to download using public-standard network protocols +a complete Transparent copy of the Document, free of added material. +If you use the latter option, you must take reasonably prudent steps, +when you begin distribution of Opaque copies in quantity, to ensure +that this Transparent copy will remain thus accessible at the stated +location until at least one year after the last time you distribute an +Opaque copy (directly or through your agents or retailers) of that +edition to the public. + +It is requested, but not required, that you contact the authors of the +Document well before redistributing any large number of copies, to give +them a chance to provide you with an updated version of the Document. + +@item +MODIFICATIONS + +You may copy and distribute a Modified Version of the Document under +the conditions of sections 2 and 3 above, provided that you release +the Modified Version under precisely this License, with the Modified +Version filling the role of the Document, thus licensing distribution +and modification of the Modified Version to whoever possesses a copy +of it. In addition, you must do these things in the Modified Version: + +@enumerate A +@item +Use in the Title Page (and on the covers, if any) a title distinct +from that of the Document, and from those of previous versions +(which should, if there were any, be listed in the History section +of the Document). You may use the same title as a previous version +if the original publisher of that version gives permission. + +@item +List on the Title Page, as authors, one or more persons or entities +responsible for authorship of the modifications in the Modified +Version, together with at least five of the principal authors of the +Document (all of its principal authors, if it has fewer than five), +unless they release you from this requirement. + +@item +State on the Title page the name of the publisher of the +Modified Version, as the publisher. + +@item +Preserve all the copyright notices of the Document. + +@item +Add an appropriate copyright notice for your modifications +adjacent to the other copyright notices. + +@item +Include, immediately after the copyright notices, a license notice +giving the public permission to use the Modified Version under the +terms of this License, in the form shown in the Addendum below. + +@item +Preserve in that license notice the full lists of Invariant Sections +and required Cover Texts given in the Document's license notice. + +@item +Include an unaltered copy of this License. + +@item +Preserve the section Entitled ``History'', Preserve its Title, and add +to it an item stating at least the title, year, new authors, and +publisher of the Modified Version as given on the Title Page. If +there is no section Entitled ``History'' in the Document, create one +stating the title, year, authors, and publisher of the Document as +given on its Title Page, then add an item describing the Modified +Version as stated in the previous sentence. + +@item +Preserve the network location, if any, given in the Document for +public access to a Transparent copy of the Document, and likewise +the network locations given in the Document for previous versions +it was based on. These may be placed in the ``History'' section. +You may omit a network location for a work that was published at +least four years before the Document itself, or if the original +publisher of the version it refers to gives permission. + +@item +For any section Entitled ``Acknowledgements'' or ``Dedications'', Preserve +the Title of the section, and preserve in the section all the +substance and tone of each of the contributor acknowledgements and/or +dedications given therein. + +@item +Preserve all the Invariant Sections of the Document, +unaltered in their text and in their titles. Section numbers +or the equivalent are not considered part of the section titles. + +@item +Delete any section Entitled ``Endorsements''. Such a section +may not be included in the Modified Version. + +@item +Do not retitle any existing section to be Entitled ``Endorsements'' or +to conflict in title with any Invariant Section. + +@item +Preserve any Warranty Disclaimers. +@end enumerate + +If the Modified Version includes new front-matter sections or +appendices that qualify as Secondary Sections and contain no material +copied from the Document, you may at your option designate some or all +of these sections as invariant. To do this, add their titles to the +list of Invariant Sections in the Modified Version's license notice. +These titles must be distinct from any other section titles. + +You may add a section Entitled ``Endorsements'', provided it contains +nothing but endorsements of your Modified Version by various +parties---for example, statements of peer review or that the text has +been approved by an organization as the authoritative definition of a +standard. + +You may add a passage of up to five words as a Front-Cover Text, and a +passage of up to 25 words as a Back-Cover Text, to the end of the list +of Cover Texts in the Modified Version. Only one passage of +Front-Cover Text and one of Back-Cover Text may be added by (or +through arrangements made by) any one entity. If the Document already +includes a cover text for the same cover, previously added by you or +by arrangement made by the same entity you are acting on behalf of, +you may not add another; but you may replace the old one, on explicit +permission from the previous publisher that added the old one. + +The author(s) and publisher(s) of the Document do not by this License +give permission to use their names for publicity for or to assert or +imply endorsement of any Modified Version. + +@item +COMBINING DOCUMENTS + +You may combine the Document with other documents released under this +License, under the terms defined in section 4 above for modified +versions, provided that you include in the combination all of the +Invariant Sections of all of the original documents, unmodified, and +list them all as Invariant Sections of your combined work in its +license notice, and that you preserve all their Warranty Disclaimers. + +The combined work need only contain one copy of this License, and +multiple identical Invariant Sections may be replaced with a single +copy. If there are multiple Invariant Sections with the same name but +different contents, make the title of each such section unique by +adding at the end of it, in parentheses, the name of the original +author or publisher of that section if known, or else a unique number. +Make the same adjustment to the section titles in the list of +Invariant Sections in the license notice of the combined work. + +In the combination, you must combine any sections Entitled ``History'' +in the various original documents, forming one section Entitled +``History''; likewise combine any sections Entitled ``Acknowledgements'', +and any sections Entitled ``Dedications''. You must delete all +sections Entitled ``Endorsements.'' + +@item +COLLECTIONS OF DOCUMENTS + +You may make a collection consisting of the Document and other documents +released under this License, and replace the individual copies of this +License in the various documents with a single copy that is included in +the collection, provided that you follow the rules of this License for +verbatim copying of each of the documents in all other respects. + +You may extract a single document from such a collection, and distribute +it individually under this License, provided you insert a copy of this +License into the extracted document, and follow this License in all +other respects regarding verbatim copying of that document. + +@item +AGGREGATION WITH INDEPENDENT WORKS + +A compilation of the Document or its derivatives with other separate +and independent documents or works, in or on a volume of a storage or +distribution medium, is called an ``aggregate'' if the copyright +resulting from the compilation is not used to limit the legal rights +of the compilation's users beyond what the individual works permit. +When the Document is included in an aggregate, this License does not +apply to the other works in the aggregate which are not themselves +derivative works of the Document. + +If the Cover Text requirement of section 3 is applicable to these +copies of the Document, then if the Document is less than one half of +the entire aggregate, the Document's Cover Texts may be placed on +covers that bracket the Document within the aggregate, or the +electronic equivalent of covers if the Document is in electronic form. +Otherwise they must appear on printed covers that bracket the whole +aggregate. + +@item +TRANSLATION + +Translation is considered a kind of modification, so you may +distribute translations of the Document under the terms of section 4. +Replacing Invariant Sections with translations requires special +permission from their copyright holders, but you may include +translations of some or all Invariant Sections in addition to the +original versions of these Invariant Sections. You may include a +translation of this License, and all the license notices in the +Document, and any Warranty Disclaimers, provided that you also include +the original English version of this License and the original versions +of those notices and disclaimers. In case of a disagreement between +the translation and the original version of this License or a notice +or disclaimer, the original version will prevail. + +If a section in the Document is Entitled ``Acknowledgements'', +``Dedications'', or ``History'', the requirement (section 4) to Preserve +its Title (section 1) will typically require changing the actual +title. + +@item +TERMINATION + +You may not copy, modify, sublicense, or distribute the Document +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense, or distribute it is void, and +will automatically terminate your rights under this License. + +However, if you cease all violation of this License, then your license +from a particular copyright holder is reinstated (a) provisionally, +unless and until the copyright holder explicitly and finally +terminates your license, and (b) permanently, if the copyright holder +fails to notify you of the violation by some reasonable means prior to +60 days after the cessation. + +Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + +Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, receipt of a copy of some or all of the same material does +not give you any rights to use it. + +@item +FUTURE REVISIONS OF THIS LICENSE + +The Free Software Foundation may publish new, revised versions +of the GNU Free Documentation License from time to time. Such new +versions will be similar in spirit to the present version, but may +differ in detail to address new problems or concerns. See +@uref{http://www.gnu.org/copyleft/}. + +Each version of the License is given a distinguishing version number. +If the Document specifies that a particular numbered version of this +License ``or any later version'' applies to it, you have the option of +following the terms and conditions either of that specified version or +of any later version that has been published (not as a draft) by the +Free Software Foundation. If the Document does not specify a version +number of this License, you may choose any version ever published (not +as a draft) by the Free Software Foundation. If the Document +specifies that a proxy can decide which future versions of this +License can be used, that proxy's public statement of acceptance of a +version permanently authorizes you to choose that version for the +Document. + +@item +RELICENSING + +``Massive Multiauthor Collaboration Site'' (or ``MMC Site'') means any +World Wide Web server that publishes copyrightable works and also +provides prominent facilities for anybody to edit those works. A +public wiki that anybody can edit is an example of such a server. A +``Massive Multiauthor Collaboration'' (or ``MMC'') contained in the +site means any set of copyrightable works thus published on the MMC +site. + +``CC-BY-SA'' means the Creative Commons Attribution-Share Alike 3.0 +license published by Creative Commons Corporation, a not-for-profit +corporation with a principal place of business in San Francisco, +California, as well as future copyleft versions of that license +published by that same organization. + +``Incorporate'' means to publish or republish a Document, in whole or +in part, as part of another Document. + +An MMC is ``eligible for relicensing'' if it is licensed under this +License, and if all works that were first published under this License +somewhere other than this MMC, and subsequently incorporated in whole +or in part into the MMC, (1) had no cover texts or invariant sections, +and (2) were thus incorporated prior to November 1, 2008. + +The operator of an MMC Site may republish an MMC contained in the site +under CC-BY-SA on the same site at any time before August 1, 2009, +provided the MMC is eligible for relicensing. + +@end enumerate + +@page +@heading ADDENDUM: How to use this License for your documents + +To use this License in a document you have written, include a copy of +the License in the document and put the following copyright and +license notices just after the title page: + +@smallexample +@group + Copyright (C) @var{year} @var{your name}. + Permission is granted to copy, distribute and/or modify this document + under the terms of the GNU Free Documentation License, Version 1.3 + or any later version published by the Free Software Foundation; + with no Invariant Sections, no Front-Cover Texts, and no Back-Cover + Texts. A copy of the license is included in the section entitled ``GNU + Free Documentation License''. +@end group +@end smallexample + +If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, +replace the ``with@dots{}Texts.'' line with this: + +@smallexample +@group + with the Invariant Sections being @var{list their titles}, with + the Front-Cover Texts being @var{list}, and with the Back-Cover Texts + being @var{list}. +@end group +@end smallexample + +If you have Invariant Sections without Cover Texts, or some other +combination of the three, merge those two alternatives to suit the +situation. + +If your document contains nontrivial examples of program code, we +recommend releasing these examples in parallel under your choice of +free software license, such as the GNU General Public License, +to permit their use in free software. + + +@c Local Variables: +@c ispell-local-pdict: "ispell-dict" +@c End: diff --git a/wisi.el b/wisi.el index d9150f3..1c4347c 100644 --- a/wisi.el +++ b/wisi.el @@ -7,8 +7,8 @@ ;; Keywords: parser ;; indentation ;; navigation -;; Version: 3.1.4 -;; package-requires: ((emacs "25.0") (seq "2.20")) +;; Version: 3.1.5 +;; package-requires: ((emacs "25.3") (seq "2.20")) ;; URL: http://stephe-leake.org/ada/wisitoken.html ;; ;; This file is part of GNU Emacs. diff --git a/wisi.info b/wisi.info deleted file mode 100644 index e24dcee..0000000 --- a/wisi.info +++ /dev/null @@ -1,1294 +0,0 @@ -This is wisi.info, produced by makeinfo version 6.7 from wisi.texi. - -Copyright (C) 1999 - 2020 Free Software Foundation, Inc. - - Permission is granted to copy, distribute and/or modify this - document under the terms of the GNU Free Documentation License, - Version 1.3 or any later version published by the Free Software - Foundation; with no Invariant Sections, with the Front-Cover texts - being "A GNU Manual", and with the Back-Cover Texts as in (a) - below. A copy of the license is included in the section entitled - "GNU Free Documentation License". - - (a) The FSF's Back-Cover Text is: "You have the freedom to copy and - modify this GNU manual. Buying copies from the FSF supports it in - developing GNU and promoting software freedom." -INFO-DIR-SECTION Emacs -START-INFO-DIR-ENTRY -* Wisi: (wisi). Error-correcting LR parsers and project integration. -END-INFO-DIR-ENTRY - - -File: wisi.info, Node: Top, Next: Overview, Up: (dir) - -Top -*** - -Wisi Version 3.1.2 - -* Menu: - -* Overview:: -* Grammar actions:: -* Project extension:: -* GNU Free Documentation License:: -* Index:: - - -File: wisi.info, Node: Overview, Next: Grammar actions, Prev: Top, Up: Top - -1 Overview -********** - -"wisi" used to be an acronym, but now it's just a name. - - The wisi package provides an elisp interface to an external parser. -It assumes the parser generator package WisiToken -(<http://stephe-leake.org/ada/wisitoken.html>, implemented in Ada), but -can use any parser that meets the same API. wisi provides several -grammar actions, to implement indentation, navigating, and syntax -highlighting (fontification). - - wisi also provides an extension to Emacs 'project.el', providing -operations useful for compilation and cross-reference. - - -File: wisi.info, Node: Grammar actions, Next: Project extension, Prev: Overview, Up: Top - -2 Grammar Actions -***************** - -Grammar actions are specified in the grammar file, in a nonterminal -declaration. We assume the user is familiar with parser grammars and -grammar actions. For example, a simple "if" statement can be declared -as: - - if_statement - : IF expression THEN statements elsif_list ELSE statements END IF SEMICOLON - %((wisi-statement-action [1 statement-start 3 motion 6 motion 10 statement-end]) - (wisi-motion-action [1 3 5 6 10]) - (wisi-indent-action [nil - [(wisi-hanging% ada-indent-broken (* 2 ada-indent-broken)) - ada-indent-broken] - nil - [ada-indent ada-indent] nil nil - [ada-indent ada-indent] nil nil nil]))% - - The item before ':' is the "left hand side", or "nonterminal". The -list of tokens after ':' is the "right hand side"; in general there can -be more than one right hand side for each nonterminal (separated by -'|'). - - The items enclosed in "%()%" are the grammar actions. They are -specified as list of elisp forms; an earlier version of the wisi package -generated an elisp parser. We keep the elisp form because it is -compact, and easier to read and write than the equivalent Ada code. The -'wisi-bnf-generate' tool converts the elisp into the required Ada -statements. - - There are two classes of actions; in-parse and post-parse. WisiToken -calls these "semantic checks" and "user actions". The in-parse actions -are done as parsing procedes; they provide extra checks that can cause -the parse to fail. Currently the only one provided is 'match-names'; it -is used to check that the declaration and end names in named Ada blocks -are the same (which can aid significantly in error correction). In the -grammar file, in-parse actions are specified in a second '%()%' block, -which can be omitted if empty. In this document, the term "action" -means "post-parse action", we use "in-parse action" unless the meaning -is clear from context. - - Executing the wisi grammar actions creates text properties in the -source file; those text properties are then used by elisp code for -various purposes. - - The text properties applied are: - -'wisi-cache' - This should be named 'wisi-navigate', but isn't for historical - reasons (there used to be only one kind of text property). - - The property contains a 'wisi-cache' object, containing: - - 'nonterm' - The nonterminal in the grammar production that specified the - action that produced this text property. - - 'token' - A token identifier naming a token in the production right hand - side containing the text this text property is applied to. - - 'last' - The position of the last character in the token, relative to - the first character (0 indexed). The text property is only - applied to the first character in the token (mostly for - historical reasons). - - 'class' - A token class; see the list of possible values in - 'wisi-statement-action' below. - - 'containing' - A marker pointing to the start of the containing token for - this token; only 'nil' for the outermost containing token in a - file. - - 'prev' - A marker pointing to the previous "motion token" in the - statement or declaration. These are normally language - keywords, but can be other things. - - 'next' - A marker pointing to the next "motion token" in the statement - or declaration. - - 'end' - A marker pointing to the end of the statement or declaration. - - wisi provides motion commands for going to the various markers. - -'wisi-name' - Contains no data, applied to a "name" of some sort. wisi provides - commands for finding the next/previous name, and returning the - text. Useful for the names of subprograms, which can then be used - to build a completion table; see - 'wisi-xref-identifier-completion-table'. - -'font-lock-face' - The standard font-lock property, specifying the face for the text. - - Some major modes do not use this for simple keywords; they use - font-lock regular expressions instead. One reason for this is so - keywords are still highlighted when the parser fails, which can - happen if there are severe syntax errors. - - Other items, like function and package names, are typically marked - with 'font-lock-face' by the parser. - -'fontified' - Another standard font-lock text property; applied whenever - 'font-lock-face' is. - -'wisi-indent' - Contains the indent (in characters) for the next line; applied to - the newline character on the preceding line. The first line in a - buffer is assumed to have indent 0. - - Each action is classified as one of 'navigate, face, indent, -in-parse'; when actions are executed, only one of the first three -classes is executed (in-parse is always executed). This reflects the -reasons the parser is run; to figure out how to go somehere (end of -current statement, start of current procedure, etc), to apply faces for -syntax highlighting, or to indent the code. - -* Menu: - -* Navigate actions:: -* Face actions:: -* Indent actions:: -* In-parse actions:: - - -File: wisi.info, Node: Navigate actions, Next: Face actions, Up: Grammar actions - -2.1 Navigate actions -==================== - -'wisi-statement-action [TOKEN CLASS ...]' - The argument is a vector; alternating items are a token index (an - integer or label indicating a token in the right hand side) and a - "token class"; one of: - - 'motion' - Create a 'wisi-cache' text property on the token, for use in a - subsequent 'wisi-motion-action'. - - 'statement-end' - Create a 'wisi-cache' text property on the token, enter a - pointer to it in the other 'wisi-cache' objects in the - statement or declaration. - - 'statement-start' - Create a 'wisi-cache' text property on the token, enter a - pointer to it in the other 'wisi-cache' objects (in the - 'containing' slot) in the statement or declaration. - - 'statement-override' - Same as 'statement-start'; marks the token to be used as the - statement start if the first token is optional. - - 'misc' - Create a 'wisi-cache' text property on the token, to be used - for some other purpose. It is good style to indicate the - purpose in a comment. - - For example, ada-mode uses a 'misc' property on left - parentheses that start a subprogram parameter list; this - distinquishes them from other left parentheses, and makes it - possible to automatically call 'ada-format-paramlist' to - format the parameter list, instead of using the standard Emacs - 'align'. - -'wisi-motion-action [TOKEN ...]' - The argument is a vector, where each element is either a token - index or a vector [INDEX ID]. - - Each terminal token must already have a 'wisi-cache' created by a - 'wisi-statement-action' (this is checked at action execution, not - during grammar generation). This action sets the 'prev, next' - slots for the chain of tokens, creating a chain of motion tokens. - - If TOKEN is a nonterminal without an ID specified, the 'wisi-cache' - must be on the first token in the nonterminal, and it is assumed to - have a valid pointer in the 'next' slot, indicating a chain of - motion tokens. That chain is linked into the chain for the current - right hand side. - - If TOKEN is a nonterminal with an ID, the region contained by the - nonterminal is searched for all 'wisi-cache' with that token ID, - and for each one where prev/next is not already set, it is linked - into the motion chain. - - Note that the "search" described here is done in the parser - process, on a tree data structure containing the data that will - eventually be stored in Emacs text properties. - -'wisi-name-action TOKEN' - TOKEN is a token index. Create a 'wisi-name' text property on the - token. - - -File: wisi.info, Node: Face actions, Next: Indent actions, Prev: Navigate actions, Up: Grammar actions - -2.2 Face actions -================ - -'wisi-face-mark-action [INDEX CLASS ...]' - The argument is a vector; alternating elements form pairs of INDEX - CLASS, where class is one of 'prefix, suffix'. - - Mark the tokens as part of a compound name, for use by later face - actions. - -'wisi-face-apply-action [TOKEN PREFIX-FACE SUFFIX-FACE ...]' - The argument is a vector; triples of items specify TOKEN, - PREFIX-FACE, SUFFIX-FACE. The faces are the elisp names of face - objects (which must declared by an '%elisp_face' declaration). - - If the token is a nonterminal, and it has been marked by a previous - 'wisi-face-mark-action', the specified faces are applied to the - prefix and suffix in the token as 'font-lock-face' text properties. - - If the token is a terminal, or a non-terminal with no face mark, - the suffix face is applied to the entire text contained by the - token. - -'wisi-face-apply-list-action [TOKEN PREFIX-FACE SUFFIX-FACE ...]' - Similar to ’wisi-face-apply-action’, but applies faces to all - tokens marked by 'wisi-face-mark-action' in each indicated - production token, and does not apply a face if there are no such - marks. - - -File: wisi.info, Node: Indent actions, Next: In-parse actions, Prev: Face actions, Up: Grammar actions - -2.3 Indent actions -================== - -Indents are computed for each line in a cumulative way as the grammar -actions are executed. Initially, each indent is set to 'nil', which -means "not computed"; this is not the same as the value '0'. The -grammar actions are executed in a bottom-up fashion; low level -productions are executed before higher level ones. In general, the -indent action for a production specifies a "delta indent"; the indent is -incremented by that amount. When all productions have been processed, -the indent has been computed for all lines. - - Indents are often given as a function call; the arguments to the -function can be other function calls, or integer expressions. -'wisitoken-bnf-generate' supports only simple integer expressions; those -using integers, integer-valued variables, parenthesis, + (plus), - -(minus), and * (multiply). - -'wisi-indent-action [INDENT ...]' - The argument is a vector, giving an indent for each token in the - production right-hand side. - - For terminals, the indents only have meaning, and are only - computed, if the token is the first on a line. For nonterminals - where the indent is not a variant of 'wisi-hanging', the indent is - only computed if the first terminal token in the nonterminal is the - first on a line. See 'wisi-hanging' in *note Indent functions:: - for the remaining case. - - An indent can have several forms. In the descriptions below, the - "current token" is given by the position of the indent expression - in the 'wisi-indent-action' argument list. - - An integer - This gives a delta indent; it is added to the total indent for - the line. - - A variable name - The name is translated to an Ada identifier by replacing "-" - with "_", and applying 'Camel_Case'. The translated name must - identify a directly visible run-time Ada integer variable; - this is checked at Ada compile time. It provides an integer - delta indent. - - For example, in Ada two indent variable names are 'ada-indent' - and 'ada-indent-broken', giving the basic ident, and the - continuation line indent. They are runtime variables so - different projects can specify them as part of a coding - standard. - - A function call - A function that computes a delta indent. See *note Indent - functions::. - - [CODE-INDENT , COMMENT-INDENT] - A vector giving separate indents for code and comments. - - Normally, the indent for trailing comments (on lines with no - code, after all code in the token) is given by the indent of - the following token in the production. When the current token - is the last, or the following tokens may be empty, or the - indent of the following token would be wrong for some reason - (for example, it is a block end), the comment indent may be - specified separately. If it is not specified, and the indent - from the next token is not available, the indent for the - current token is used for code and comments. - - Comment lines that are not trailing are indented by - CODE-INDENT. - - (label . INDENT) - If token labels are used in a right hand side, they must be - given explicitly in the indent arguments, using he lisp "cons" - syntax. Labels are normally only used with EBNF grammars, - which expand into multiple right hand sides, with optional - tokens simply left out. Explicit labels on the indent - arguments allow them to be left out as well. - -* Menu: - -* Indent functions:: -* Indent example:: - - -File: wisi.info, Node: Indent functions, Next: Indent example, Up: Indent actions - -2.3.1 Indent functions ----------------------- - -'wisi-anchored TOKEN OFFSET' - Sets the indent for the current token to be OFFSET (an integer - expression) from the start of TOKEN (a token index); the current - token is "anchored to" TOKEN. - -'wisi-anchored* TOKEN OFFSET' - Sets the indent for the current token to be OFFSET from the start - of TOKEN, but only if TOKEN is the first token on a line; otherwise - no indent - -'wisi-anchored*- TOKEN OFFSET' - Sets the indent for the current token to be OFFSET from the start - of TOKEN, but only if TOKEN is the first token on a line and the - indent for the current token accumulated so far is nil. - -'wisi-anchored% TOKEN OFFSET' - If there is an opening parenthesis containing TOKEN in the line - containing TOKEN, set the current indent to OFFSET from that - parenthesis. Otherwise, OFFSET gives an indent delta. - -'wisi-anchored%- TOKEN OFFSET' - Same as 'wisi-anchored%', but only if the current token accumulated - indent is nil. - -'wisi-hanging DELTA-1 DELTA-2' - The current token is assumed to be a nonterminal. If the text it - contains spans multiple lines, use DELTA-1 for the first line, - DELTA-2 for the rest. If the current token is only on one line, - use DELTA-1. - - DELTA-1 and DELTA-2 can be any IDENT expression, except a variant - of 'wisi-hanging'. - -'wisi-hanging% DELTA-1 DELTA-2' - Similar to 'wisi-hanging'; if the first terminal token in the - current nonterminal is the first token on the first line, use - DELTA-1 for the first line and DELTA-2 for the rest. Otherwise, - use DELTA-1 for all lines. - -'wisi-hanging%- DELTA-1 DELTA-2' - Same as 'wisi-hanging%', except applied only if the current token - accumulated indent is nil. - -'Language-specific function' - Language-specific indent functions are specified by an - '%elisp_indent' declaration in the grammar file. Each function - specifies how many arguments it accepts; this is checked at action - runtime, not during grammar generation. Each argument is an INDENT - as described above, or a token ID prefixed by ''' (to allow - distinguishing token IDs from variable names). - - -File: wisi.info, Node: Indent example, Prev: Indent functions, Up: Indent actions - -2.3.2 Indent example --------------------- - -The example 'if_statement' grammar nonterminal is: - - if_statement - : IF expression THEN statements elsif_list ELSE statements END IF SEMICOLON - %((wisi-indent-action [nil - [(wisi-hanging% ada-indent-broken (* 2 ada-indent-broken)) - ada-indent-broken] - nil - [ada-indent ada-indent] nil nil - [ada-indent ada-indent] nil nil nil]))% - - We trace how the indent is computed for this sample Ada code: - - 1: if A < B and - 2: C < D - 3: -- comment on expression - 4: then - 5: if E then - 6: Do_E; - 7: -- comment on statement - 8: elsif F then - 9: G := A + Compute_Something - 10: (arg_1, arg_2); - 11: end if; - 12: end if; - - First, the indent for the lower-level nonterminals ('expression, -statements, elsif_list') are computed. Assume they set the indent for -line 10 to 2 (for the hanging expression) and leave the rest at nil. - - Next, the action for the inner 'if_statement' is executed. Most of -the tokens specify an indent of 'nil', which means the current -accumulated indent is not changed. For the others, the action is as -follows: - -'expression:' - The expression 'E' is contained on one line, and it is not the - first token on that line, so the indent for line 5 is not changed. - -'statements: [ada-indent ada-indent]' - This specifies separate indents for code and trailing comments, - because otherwise the trailing comments would be indented with the - following 'THEN'; instead they are indented with the expression - code; see the comment on line 7. - - Here 'ada-indent' is 3, so the indent for lines 6 and 7 (for the - first occurence of 'statments') is incremented from 'nil' to '3'. - - For the second occurence of 'statements', line 9 is incremented - from 'nil' to '3', and line 10 from '2' to '5'. - - At this point, the accumulated indents are (the indent is given after -the line number): - 1: nil : if A < B and - 2: nil : C < D - 3: nil : -- comment on expression - 4: nil : then - 5: nil : if E then - 6: 3 : Do_E; - 7: 3 : -- comment on statement - 8: nil : elsif F then - 9: 3 : G := A + Compute_Something - 10: 5 : (arg_1, arg_2); - 11: nil : end if; - 12: nil : end if; - - Then the action is executed for the outer 'if_statement': - -'expression: [(wisi-hanging% ada-indent-broken (* 2 ada-indent-broken)) ada-indent-broken]' - This specifies separate indents for code and trailing comments, - because otherwise the trailing comments would be indented with the - following 'THEN'; instead they are indented with the expression - code; see the comment on line 3. - - In this case, 'wisi-hanging%' returns DELTA-1, which is - 'ada-indent-broken', which is 2. So the indent for line 2 is - incremented from 'nil' to '2'. - - The indent for line 3 is also incremented from 'nil' to '2'. - -'statements: [ada-indent ada-indent]' - Here there is only one statement; the nested 'if_statement'. The - indent for lines 5 .. 11 are each incremented by 3. - - The final result is: - 1: nil : if A < B and - 2: 2 : C < D - 3: 2 : -- comment on expression - 4: nil : then - 5: 3 : if E then - 6: 6 : Do_E; - 7: 6 : -- comment on statement - 8: 3 : elsif F then - 9: 6 : G := A + Compute_Something - 10: 8 : (arg_1, arg_2); - 11: 6 : end if; - 12: nil : end if; - - In a full grammar, the top production should specify an indent of 0, -not nil, for tokens that are not indented; then every line will have a -non-nil indent. However, in normal operation a nil indent is treated as -0; the 'wisi-indent' text property is not set for lines that have nil -indent, and 'wisi-indent-region' detects that and uses 0 for the indent. -You can set the variable 'wisi-debug' to a value > 0 to signal an error -for nil indents; this is useful to catch indent errors during grammar -development. - - -File: wisi.info, Node: In-parse actions, Prev: Indent actions, Up: Grammar actions - -2.4 In-parse actions -==================== - -'wisi-propagate-name TOKEN' - The argument is a token index. Set the 'name' component of the - left-hand-side parse-time token object to the 'name' component of - the identified token, if it is not empty. Otherwise use the - 'byte_region' component. - -'wisi-merge-name FIRST-TOKEN, LAST-TOKEN' - The arguments are token indices, giving a range of tokens. - LAST-TOKEN may be omitted if it is the same as FIRST-TOKEN. - - Set the 'name' component of the left-hand-side to the merger of the - 'name' or 'byte-region' components of the identified tokens. - -'wisi-match-name START-TOKEN END-TOKEN' - The arguments are token indices. Compare the text contained by the - 'name' (or 'byte_region' if 'name' is empty) token components for - START-TOKEN and END-TOKEN; signal a parse error if they are - different. - - The behavior when a name is missing is determined by the runtime - language variable given in the '%end_names_optional_option' - declaration; if True, a missing name that is supposed to match a - present name is an error. Both names missing is not an error - (assuming that is allowed by the grammar). - - -File: wisi.info, Node: Project extension, Next: GNU Free Documentation License, Prev: Grammar actions, Up: Top - -3 Project extension -******************* - -wisi defines the 'cl-defstuct' 'wisi-prj', with operations suitable for -compilation and cross-reference. - - In order to use wisi projects, the user must write project files and -customize 'project-find-functions' and 'xref-backend-functions'. - -* Menu: - -* Project files:: -* Selecting projects:: -* Casing exception files:: -* Other project functions:: - - -File: wisi.info, Node: Project files, Next: Selecting projects, Up: Project extension - -3.1 Project files -================= - -Project file names must have an extension given by -'wisi-prj-file-extensions' (default '.adp, .prj'). - - Project files have a simple syntax; they may be edited directly. -Each line specifies a project variable name and its value, separated by -"=": - - src_dir=/Projects/my_project/src_1 - src_dir=/Projects/my_project/src_2 - - There must be no space between the variable name and "=", and no -trailing spaces after the value. - - Any line that does not have an "=" is a comment. - - Some variables (like 'src_dir') are lists; each line in the project -file specifies one element of the list. The value on the last line is -the last element in the list. - - A variable name that starts with '$' is set as a process environment -variable, for processes launched from Emacs for the project. - - In values, process environment variables can be referenced using the -normal '$var' syntax. - - In values, relative file names are expanded relative to the directory -containing the project file. - - Here is the list of project variables defined by wisi; major modes -may add more. - -'casing' [slot: 'case-exception-files'] - List of files containing casing exceptions. *Note Casing exception - files::. - -'src_dir' [slot: 'source-path'] - A list of directories to search for source files. - - -File: wisi.info, Node: Selecting projects, Next: Casing exception files, Prev: Project files, Up: Project extension - -3.2 Selecting projects -====================== - -The current project can either be indicated by a global variable (called -a "selected project"), or depend on the current buffer. - - In addition, the project file can be parsed each time it is needed, -or the result cached to improve response time, - - One reason to use a selected project is to handle a hierarchy of -projects; if projects B and C both depend on library project A, then -when in a file of project A, there is no way to determine which of the -three projects to return. So the user must indicate which is active, by -using one of 'wisi-prj-select-file' or 'wisi-prj-select-cache'. - - In addition, if changing from one project to another requires setting -global resources that must also be unset (such as a syntax propertize -hook or compilation filter hook), then the project should define -'wisi-prj-deselect' in addition to 'wisi-prj-select'. Such projects -require having a selected current project, so it can be deselected -before a new one is selected. One example of such projects is ada-mode. - - One way to declare each project is to add a Local Variables section -in the main Makefile for the project; when the Makefile is first -visited, the project is declared. In the examples here, we assume that -approach is used; each gives an :eval line. - - Note that 'wisi-prj-current-parse' and 'wisi-prj-current-cached' -always succeed after some project is selected; no functions after them -on 'project-find-functions' will be called. That's why the depth is 90 -for those in the examples. - -No caching, current project depends on current buffer - - (add-hook 'project-find-functions #'wisi-prj-find-dominating-parse 0) - - :eval (wisi-prj-set-dominating "foo.prj" (foo-prj-default "prj-name")) - - 'wisi-prj-set-dominating' declares the name of a project file with - a default project object, and ensures that the current buffer file - name is in 'wisi-prj--dominating'. - - 'wisi-prj-find-dominating-parse' looks for the filenames in - 'wisi-prj--dominiating' in the parent directories of the current - buffer. When one is found, the associated project file is parsed, - using the default project object to dispatch to the appropriate - parsers. Then the final project object is returned. - -Caching, current project depends on current buffer - - (add-hook 'project-find-functions #'wisi-prj-find-dominating-cached 0) - - :eval (wisi-prj-cache-dominating "foo.prj" (foo-prj-default "prj-name")) - - 'wisi-prj-cache-dominating' declares the project file, parses it, - and saves the project object in a cache indexed by the absolute - project file name. - - 'wisi-prj-find-dominating-cached' finds the dominating project - file, and retrieves the object from the cache. - -No caching, last selected project is current - - (add-hook 'project-find-functions #'wisi-prj-current-parse 90) - - :eval: (wisi-prj-select-file <prj-file> (foo-prj-default "prj-name")) - - 'wisi-prj-select-file' sets the project file as the current - project, and saves the default project object. - - 'wisi-prj-current-parse' parses the current project file, using the - saved default project object, and returns the project object. - -Caching, last selected project is current - - (add-hook 'project-find-functions #'wisi-prj-current-cached 90) - - :eval: (wisi-prj-select-cache <prj-file> (foo-prj-default "prj-name")) - - 'wisi-prj-select-cache' parses the project file, caches the project - object. - - 'wisi-prj-current-cached' returns the cached current project - object. - - In addition, the user should set 'xref-backend-functions'. -Currently, there is only one choice for wisi projects: - - (add-to-list 'xref-backend-functions #'wisi-prj-xref-backend 90) - - 'wisi-prj-xref-backend' returns the current wisi project object. - - -File: wisi.info, Node: Casing exception files, Next: Other project functions, Prev: Selecting projects, Up: Project extension - -3.3 Casing exception files -========================== - -Each line in a case exception file specifies the casing of one word or -word fragment. If an exception is defined in multiple files, the first -occurrence is used. - - If the word starts with an asterisk ('*'), it defines the casing of a -word fragment (or "substring"); part of a word between two underscores -or word boundary. - - For example: - - DOD - *IO - GNAT - - The word fragment '*IO' applies to any word containing "_io"; -'Text_IO', 'Hardware_IO', etc. - - -File: wisi.info, Node: Other project functions, Prev: Casing exception files, Up: Project extension - -3.4 Other project functions -=========================== - -'wisi-refresh-prj-cache (not-full)' - Refreshes all cached data in the project, and re-selects the - project. If NOT-FULL is non-nil, slow refresh operations are - skipped. - - This reparses the project file, and any cross reference - information. - -'wisi-prj-select-dominating (dominating-file)' - Find a wisi-prj matching DOMINATING-FILE (defaults to the current - buffer file). If the associated project is current, do nothing. - If it is not current, select it. - - This is useful before running 'compilation-start', to ensure the - correct project is current. - - -File: wisi.info, Node: GNU Free Documentation License, Next: Index, Prev: Project extension, Up: Top - -Appendix A GNU Free Documentation License -***************************************** - - Version 1.3, 3 November 2008 - - Copyright (C) 2000, 2001, 2002, 2007, 2008, 2009 Free Software Foundation, Inc. - <http://fsf.org/> - - Everyone is permitted to copy and distribute verbatim copies - of this license document, but changing it is not allowed. - - 0. PREAMBLE - - The purpose of this License is to make a manual, textbook, or other - functional and useful document "free" in the sense of freedom: to - assure everyone the effective freedom to copy and redistribute it, - with or without modifying it, either commercially or - noncommercially. Secondarily, this License preserves for the - author and publisher a way to get credit for their work, while not - being considered responsible for modifications made by others. - - This License is a kind of "copyleft", which means that derivative - works of the document must themselves be free in the same sense. - It complements the GNU General Public License, which is a copyleft - license designed for free software. - - We have designed this License in order to use it for manuals for - free software, because free software needs free documentation: a - free program should come with manuals providing the same freedoms - that the software does. But this License is not limited to - software manuals; it can be used for any textual work, regardless - of subject matter or whether it is published as a printed book. We - recommend this License principally for works whose purpose is - instruction or reference. - - 1. APPLICABILITY AND DEFINITIONS - - This License applies to any manual or other work, in any medium, - that contains a notice placed by the copyright holder saying it can - be distributed under the terms of this License. Such a notice - grants a world-wide, royalty-free license, unlimited in duration, - to use that work under the conditions stated herein. The - "Document", below, refers to any such manual or work. Any member - of the public is a licensee, and is addressed as "you". You accept - the license if you copy, modify or distribute the work in a way - requiring permission under copyright law. - - A "Modified Version" of the Document means any work containing the - Document or a portion of it, either copied verbatim, or with - modifications and/or translated into another language. - - A "Secondary Section" is a named appendix or a front-matter section - of the Document that deals exclusively with the relationship of the - publishers or authors of the Document to the Document's overall - subject (or to related matters) and contains nothing that could - fall directly within that overall subject. (Thus, if the Document - is in part a textbook of mathematics, a Secondary Section may not - explain any mathematics.) The relationship could be a matter of - historical connection with the subject or with related matters, or - of legal, commercial, philosophical, ethical or political position - regarding them. - - The "Invariant Sections" are certain Secondary Sections whose - titles are designated, as being those of Invariant Sections, in the - notice that says that the Document is released under this License. - If a section does not fit the above definition of Secondary then it - is not allowed to be designated as Invariant. The Document may - contain zero Invariant Sections. If the Document does not identify - any Invariant Sections then there are none. - - The "Cover Texts" are certain short passages of text that are - listed, as Front-Cover Texts or Back-Cover Texts, in the notice - that says that the Document is released under this License. A - Front-Cover Text may be at most 5 words, and a Back-Cover Text may - be at most 25 words. - - A "Transparent" copy of the Document means a machine-readable copy, - represented in a format whose specification is available to the - general public, that is suitable for revising the document - straightforwardly with generic text editors or (for images composed - of pixels) generic paint programs or (for drawings) some widely - available drawing editor, and that is suitable for input to text - formatters or for automatic translation to a variety of formats - suitable for input to text formatters. A copy made in an otherwise - Transparent file format whose markup, or absence of markup, has - been arranged to thwart or discourage subsequent modification by - readers is not Transparent. An image format is not Transparent if - used for any substantial amount of text. A copy that is not - "Transparent" is called "Opaque". - - Examples of suitable formats for Transparent copies include plain - ASCII without markup, Texinfo input format, LaTeX input format, - SGML or XML using a publicly available DTD, and standard-conforming - simple HTML, PostScript or PDF designed for human modification. - Examples of transparent image formats include PNG, XCF and JPG. - Opaque formats include proprietary formats that can be read and - edited only by proprietary word processors, SGML or XML for which - the DTD and/or processing tools are not generally available, and - the machine-generated HTML, PostScript or PDF produced by some word - processors for output purposes only. - - The "Title Page" means, for a printed book, the title page itself, - plus such following pages as are needed to hold, legibly, the - material this License requires to appear in the title page. For - works in formats which do not have any title page as such, "Title - Page" means the text near the most prominent appearance of the - work's title, preceding the beginning of the body of the text. - - The "publisher" means any person or entity that distributes copies - of the Document to the public. - - A section "Entitled XYZ" means a named subunit of the Document - whose title either is precisely XYZ or contains XYZ in parentheses - following text that translates XYZ in another language. (Here XYZ - stands for a specific section name mentioned below, such as - "Acknowledgements", "Dedications", "Endorsements", or "History".) - To "Preserve the Title" of such a section when you modify the - Document means that it remains a section "Entitled XYZ" according - to this definition. - - The Document may include Warranty Disclaimers next to the notice - which states that this License applies to the Document. These - Warranty Disclaimers are considered to be included by reference in - this License, but only as regards disclaiming warranties: any other - implication that these Warranty Disclaimers may have is void and - has no effect on the meaning of this License. - - 2. VERBATIM COPYING - - You may copy and distribute the Document in any medium, either - commercially or noncommercially, provided that this License, the - copyright notices, and the license notice saying this License - applies to the Document are reproduced in all copies, and that you - add no other conditions whatsoever to those of this License. You - may not use technical measures to obstruct or control the reading - or further copying of the copies you make or distribute. However, - you may accept compensation in exchange for copies. If you - distribute a large enough number of copies you must also follow the - conditions in section 3. - - You may also lend copies, under the same conditions stated above, - and you may publicly display copies. - - 3. COPYING IN QUANTITY - - If you publish printed copies (or copies in media that commonly - have printed covers) of the Document, numbering more than 100, and - the Document's license notice requires Cover Texts, you must - enclose the copies in covers that carry, clearly and legibly, all - these Cover Texts: Front-Cover Texts on the front cover, and - Back-Cover Texts on the back cover. Both covers must also clearly - and legibly identify you as the publisher of these copies. The - front cover must present the full title with all words of the title - equally prominent and visible. You may add other material on the - covers in addition. Copying with changes limited to the covers, as - long as they preserve the title of the Document and satisfy these - conditions, can be treated as verbatim copying in other respects. - - If the required texts for either cover are too voluminous to fit - legibly, you should put the first ones listed (as many as fit - reasonably) on the actual cover, and continue the rest onto - adjacent pages. - - If you publish or distribute Opaque copies of the Document - numbering more than 100, you must either include a machine-readable - Transparent copy along with each Opaque copy, or state in or with - each Opaque copy a computer-network location from which the general - network-using public has access to download using public-standard - network protocols a complete Transparent copy of the Document, free - of added material. If you use the latter option, you must take - reasonably prudent steps, when you begin distribution of Opaque - copies in quantity, to ensure that this Transparent copy will - remain thus accessible at the stated location until at least one - year after the last time you distribute an Opaque copy (directly or - through your agents or retailers) of that edition to the public. - - It is requested, but not required, that you contact the authors of - the Document well before redistributing any large number of copies, - to give them a chance to provide you with an updated version of the - Document. - - 4. MODIFICATIONS - - You may copy and distribute a Modified Version of the Document - under the conditions of sections 2 and 3 above, provided that you - release the Modified Version under precisely this License, with the - Modified Version filling the role of the Document, thus licensing - distribution and modification of the Modified Version to whoever - possesses a copy of it. In addition, you must do these things in - the Modified Version: - - A. Use in the Title Page (and on the covers, if any) a title - distinct from that of the Document, and from those of previous - versions (which should, if there were any, be listed in the - History section of the Document). You may use the same title - as a previous version if the original publisher of that - version gives permission. - - B. List on the Title Page, as authors, one or more persons or - entities responsible for authorship of the modifications in - the Modified Version, together with at least five of the - principal authors of the Document (all of its principal - authors, if it has fewer than five), unless they release you - from this requirement. - - C. State on the Title page the name of the publisher of the - Modified Version, as the publisher. - - D. Preserve all the copyright notices of the Document. - - E. Add an appropriate copyright notice for your modifications - adjacent to the other copyright notices. - - F. Include, immediately after the copyright notices, a license - notice giving the public permission to use the Modified - Version under the terms of this License, in the form shown in - the Addendum below. - - G. Preserve in that license notice the full lists of Invariant - Sections and required Cover Texts given in the Document's - license notice. - - H. Include an unaltered copy of this License. - - I. Preserve the section Entitled "History", Preserve its Title, - and add to it an item stating at least the title, year, new - authors, and publisher of the Modified Version as given on the - Title Page. If there is no section Entitled "History" in the - Document, create one stating the title, year, authors, and - publisher of the Document as given on its Title Page, then add - an item describing the Modified Version as stated in the - previous sentence. - - J. Preserve the network location, if any, given in the Document - for public access to a Transparent copy of the Document, and - likewise the network locations given in the Document for - previous versions it was based on. These may be placed in the - "History" section. You may omit a network location for a work - that was published at least four years before the Document - itself, or if the original publisher of the version it refers - to gives permission. - - K. For any section Entitled "Acknowledgements" or "Dedications", - Preserve the Title of the section, and preserve in the section - all the substance and tone of each of the contributor - acknowledgements and/or dedications given therein. - - L. Preserve all the Invariant Sections of the Document, unaltered - in their text and in their titles. Section numbers or the - equivalent are not considered part of the section titles. - - M. Delete any section Entitled "Endorsements". Such a section - may not be included in the Modified Version. - - N. Do not retitle any existing section to be Entitled - "Endorsements" or to conflict in title with any Invariant - Section. - - O. Preserve any Warranty Disclaimers. - - If the Modified Version includes new front-matter sections or - appendices that qualify as Secondary Sections and contain no - material copied from the Document, you may at your option designate - some or all of these sections as invariant. To do this, add their - titles to the list of Invariant Sections in the Modified Version's - license notice. These titles must be distinct from any other - section titles. - - You may add a section Entitled "Endorsements", provided it contains - nothing but endorsements of your Modified Version by various - parties--for example, statements of peer review or that the text - has been approved by an organization as the authoritative - definition of a standard. - - You may add a passage of up to five words as a Front-Cover Text, - and a passage of up to 25 words as a Back-Cover Text, to the end of - the list of Cover Texts in the Modified Version. Only one passage - of Front-Cover Text and one of Back-Cover Text may be added by (or - through arrangements made by) any one entity. If the Document - already includes a cover text for the same cover, previously added - by you or by arrangement made by the same entity you are acting on - behalf of, you may not add another; but you may replace the old - one, on explicit permission from the previous publisher that added - the old one. - - The author(s) and publisher(s) of the Document do not by this - License give permission to use their names for publicity for or to - assert or imply endorsement of any Modified Version. - - 5. COMBINING DOCUMENTS - - You may combine the Document with other documents released under - this License, under the terms defined in section 4 above for - modified versions, provided that you include in the combination all - of the Invariant Sections of all of the original documents, - unmodified, and list them all as Invariant Sections of your - combined work in its license notice, and that you preserve all - their Warranty Disclaimers. - - The combined work need only contain one copy of this License, and - multiple identical Invariant Sections may be replaced with a single - copy. If there are multiple Invariant Sections with the same name - but different contents, make the title of each such section unique - by adding at the end of it, in parentheses, the name of the - original author or publisher of that section if known, or else a - unique number. Make the same adjustment to the section titles in - the list of Invariant Sections in the license notice of the - combined work. - - In the combination, you must combine any sections Entitled - "History" in the various original documents, forming one section - Entitled "History"; likewise combine any sections Entitled - "Acknowledgements", and any sections Entitled "Dedications". You - must delete all sections Entitled "Endorsements." - - 6. COLLECTIONS OF DOCUMENTS - - You may make a collection consisting of the Document and other - documents released under this License, and replace the individual - copies of this License in the various documents with a single copy - that is included in the collection, provided that you follow the - rules of this License for verbatim copying of each of the documents - in all other respects. - - You may extract a single document from such a collection, and - distribute it individually under this License, provided you insert - a copy of this License into the extracted document, and follow this - License in all other respects regarding verbatim copying of that - document. - - 7. AGGREGATION WITH INDEPENDENT WORKS - - A compilation of the Document or its derivatives with other - separate and independent documents or works, in or on a volume of a - storage or distribution medium, is called an "aggregate" if the - copyright resulting from the compilation is not used to limit the - legal rights of the compilation's users beyond what the individual - works permit. When the Document is included in an aggregate, this - License does not apply to the other works in the aggregate which - are not themselves derivative works of the Document. - - If the Cover Text requirement of section 3 is applicable to these - copies of the Document, then if the Document is less than one half - of the entire aggregate, the Document's Cover Texts may be placed - on covers that bracket the Document within the aggregate, or the - electronic equivalent of covers if the Document is in electronic - form. Otherwise they must appear on printed covers that bracket - the whole aggregate. - - 8. TRANSLATION - - Translation is considered a kind of modification, so you may - distribute translations of the Document under the terms of section - 4. Replacing Invariant Sections with translations requires special - permission from their copyright holders, but you may include - translations of some or all Invariant Sections in addition to the - original versions of these Invariant Sections. You may include a - translation of this License, and all the license notices in the - Document, and any Warranty Disclaimers, provided that you also - include the original English version of this License and the - original versions of those notices and disclaimers. In case of a - disagreement between the translation and the original version of - this License or a notice or disclaimer, the original version will - prevail. - - If a section in the Document is Entitled "Acknowledgements", - "Dedications", or "History", the requirement (section 4) to - Preserve its Title (section 1) will typically require changing the - actual title. - - 9. TERMINATION - - You may not copy, modify, sublicense, or distribute the Document - except as expressly provided under this License. Any attempt - otherwise to copy, modify, sublicense, or distribute it is void, - and will automatically terminate your rights under this License. - - However, if you cease all violation of this License, then your - license from a particular copyright holder is reinstated (a) - provisionally, unless and until the copyright holder explicitly and - finally terminates your license, and (b) permanently, if the - copyright holder fails to notify you of the violation by some - reasonable means prior to 60 days after the cessation. - - Moreover, your license from a particular copyright holder is - reinstated permanently if the copyright holder notifies you of the - violation by some reasonable means, this is the first time you have - received notice of violation of this License (for any work) from - that copyright holder, and you cure the violation prior to 30 days - after your receipt of the notice. - - Termination of your rights under this section does not terminate - the licenses of parties who have received copies or rights from you - under this License. If your rights have been terminated and not - permanently reinstated, receipt of a copy of some or all of the - same material does not give you any rights to use it. - - 10. FUTURE REVISIONS OF THIS LICENSE - - The Free Software Foundation may publish new, revised versions of - the GNU Free Documentation License from time to time. Such new - versions will be similar in spirit to the present version, but may - differ in detail to address new problems or concerns. See - <http://www.gnu.org/copyleft/>. - - Each version of the License is given a distinguishing version - number. If the Document specifies that a particular numbered - version of this License "or any later version" applies to it, you - have the option of following the terms and conditions either of - that specified version or of any later version that has been - published (not as a draft) by the Free Software Foundation. If the - Document does not specify a version number of this License, you may - choose any version ever published (not as a draft) by the Free - Software Foundation. If the Document specifies that a proxy can - decide which future versions of this License can be used, that - proxy's public statement of acceptance of a version permanently - authorizes you to choose that version for the Document. - - 11. RELICENSING - - "Massive Multiauthor Collaboration Site" (or "MMC Site") means any - World Wide Web server that publishes copyrightable works and also - provides prominent facilities for anybody to edit those works. A - public wiki that anybody can edit is an example of such a server. - A "Massive Multiauthor Collaboration" (or "MMC") contained in the - site means any set of copyrightable works thus published on the MMC - site. - - "CC-BY-SA" means the Creative Commons Attribution-Share Alike 3.0 - license published by Creative Commons Corporation, a not-for-profit - corporation with a principal place of business in San Francisco, - California, as well as future copyleft versions of that license - published by that same organization. - - "Incorporate" means to publish or republish a Document, in whole or - in part, as part of another Document. - - An MMC is "eligible for relicensing" if it is licensed under this - License, and if all works that were first published under this - License somewhere other than this MMC, and subsequently - incorporated in whole or in part into the MMC, (1) had no cover - texts or invariant sections, and (2) were thus incorporated prior - to November 1, 2008. - - The operator of an MMC Site may republish an MMC contained in the - site under CC-BY-SA on the same site at any time before August 1, - 2009, provided the MMC is eligible for relicensing. - -ADDENDUM: How to use this License for your documents -==================================================== - -To use this License in a document you have written, include a copy of -the License in the document and put the following copyright and license -notices just after the title page: - - Copyright (C) YEAR YOUR NAME. - Permission is granted to copy, distribute and/or modify this document - under the terms of the GNU Free Documentation License, Version 1.3 - or any later version published by the Free Software Foundation; - with no Invariant Sections, no Front-Cover Texts, and no Back-Cover - Texts. A copy of the license is included in the section entitled ``GNU - Free Documentation License''. - - If you have Invariant Sections, Front-Cover Texts and Back-Cover -Texts, replace the "with...Texts." line with this: - - with the Invariant Sections being LIST THEIR TITLES, with - the Front-Cover Texts being LIST, and with the Back-Cover Texts - being LIST. - - If you have Invariant Sections without Cover Texts, or some other -combination of the three, merge those two alternatives to suit the -situation. - - If your document contains nontrivial examples of program code, we -recommend releasing these examples in parallel under your choice of free -software license, such as the GNU General Public License, to permit -their use in free software. - - -File: wisi.info, Node: Index, Prev: GNU Free Documentation License, Up: Top - -Index -***** - - - -Tag Table: -Node: Top920 -Node: Overview1119 -Node: Grammar actions1761 -Node: Navigate actions7174 -Node: Face actions10027 -Node: Indent actions11338 -Node: Indent functions15148 -Node: Indent example17443 -Node: In-parse actions21781 -Node: Project extension23077 -Node: Project files23586 -Node: Selecting projects25006 -Node: Casing exception files29008 -Node: Other project functions29666 -Node: GNU Free Documentation License30424 -Node: Index55580 - -End Tag Table - - -Local Variables: -coding: utf-8 -End: diff --git a/wisi.texi b/wisi.texi index 8028810..ac1be79 100644 --- a/wisi.texi +++ b/wisi.texi @@ -25,7 +25,7 @@ developing GNU and promoting software freedom.'' @titlepage @sp 10 -@title Wisi Version 3.1.3 +@title Wisi Version 3.1.5 @page @vskip 0pt plus 1filll @insertcopying @@ -37,7 +37,7 @@ developing GNU and promoting software freedom.'' @node Top @top Top -Wisi Version 3.1.3 +Wisi Version 3.1.5 @end ifnottex @menu diff --git a/wisitoken-user_guide.info b/wisitoken-user_guide.info deleted file mode 100644 index 7bbdb1b..0000000 --- a/wisitoken-user_guide.info +++ /dev/null @@ -1,611 +0,0 @@ -This is wisitoken-user_guide.info, produced by makeinfo version 6.7 from -wisitoken-user_guide.texinfo. - -Copyright (C) 2014-2015, 2017, 2018, 2020 Stephen Leake. - - Permission is granted to copy, distribute and/or modify this - document under the terms of the GNU Free Documentation License, - Version 1.3 or any later version published by the Free Software - Foundation; with no Invariant Sections, no Front-Cover Texts and no - Back-Cover Texts. A copy of the license is included in the section - entitled "GNU Free Documentation License". -INFO-DIR-SECTION Parser generators -START-INFO-DIR-ENTRY -* wisitoken-bnf-generate: (wisitoken-bnf-generate). Ada and Elisp parser generator -END-INFO-DIR-ENTRY - - -File: wisitoken-user_guide.info, Node: Top, Next: Overview, Up: (dir) - -WisiToken User Guide -******************** - -Copyright (C) 2014-2015, 2017, 2018, 2020 Stephen Leake. - - Permission is granted to copy, distribute and/or modify this - document under the terms of the GNU Free Documentation License, - Version 1.3 or any later version published by the Free Software - Foundation; with no Invariant Sections, no Front-Cover Texts and no - Back-Cover Texts. A copy of the license is included in the section - entitled "GNU Free Documentation License". - -* Menu: - -* Overview:: -* Common grammar problems:: -* Grammar File Syntax:: - - -File: wisitoken-user_guide.info, Node: Overview, Next: Common grammar problems, Prev: Top, Up: Top - -1 Overview -********** - -WisiToken is a parser and parser generator toolkit, supporting -generalized LR (both LALR and LR1) and packrat parsers; the LR parser -provides robust error recovery. The grammar can be expressed as either -Ada source code statements, or in an EBNF file. The parser generator -generates Ada, either plain or assuming the Emacs wisi package. - - At one point, "wisi" was short for "Wisent Indentation engine"; the -Emacs 'wisi' package implements an indentation engine that used to be -based on the Emacs wisent parser. However, that parser has now been -replaced by the WisiToken parser, so "wisi" is just a name. - -* Menu: - -* Install:: - - -File: wisitoken-user_guide.info, Node: Install, Up: Overview - -1.1 Install -=========== - -WisiToken is available as source code only, although a subset is -available in the GNU ELPA package 'wisi'. - - You will also need to install a lexer generator. WisiToken supports -re2c, and other lexers can be added. - - re2c is available from <http://re2c.org/>; it is also packaged in -Mingw64 and Debian. WisiToken requires at least version 1.3. WisiToken -assumes the executable 're2c' is in '$PATH'. - - -File: wisitoken-user_guide.info, Node: Common grammar problems, Next: Grammar File Syntax, Prev: Overview, Up: Top - -2 Common grammar problems -************************* - -LALR grammars are tricky. Here we describe some common problems people -run into. - -* Menu: - -* Empty choice in list:: - - -File: wisitoken-user_guide.info, Node: Empty choice in list, Up: Common grammar problems - -2.1 Empty choice in list -======================== - -Many programming languages have lists in the grammar. For example, Ada -has lists of declarations: - - package_body - : PACKAGE name IS declaration_list BEGIN statement_list END SEMICOLON - ; - - declaration_list - : declaration - | declaration_list declaration - ; - - declaration - : object_declaration - | subprogram_declaration - ;; ... - ; - - Note that the above grammar fragment does not allow an empty -declaration_list. But Ada does, so the question is how can we add that -to the grammar. - - There are four choices: - - 1. Add an empty declaration choice to declaration_list: - - declaration_list - : ;; empty list - | declaration - | declaration_list declaration - ; - This is now redundant; since declaration_list can be empty, the - second choice is not needed: - declaration_list - : ;; empty list - | declaration_list declaration - ; - - 2. Add an empty declaration choice to declaration: - - declaration - : ;; empty declaration - | object_declaration - | subprogram_declaration - ;; ... - ; - - 3. Add another rule with the empty production: - - package_body - : PACKAGE name IS declarative_part BEGIN statement_list END SEMICOLON - ; - - declarative_part - : ;; empty - | declaration_list - ; - - declaration_list - : declaration - | declaration_list declaration - ; - - declaration - : object_declaration - | subprogram_declaration - ;; ... - ; - - 4. Add another choice in package_body that leaves out - declaration_list: - package_body - : PACKAGE name IS declaration_list BEGIN statement_list END SEMICOLON - | PACKAGE name IS BEGIN statement_list END SEMICOLON - ; - - Choice 1 is redundant, giving parse errors at parse time. Consider -the following statements, where "<empty>" is used to indicate an empty -declaration: - - 1) package One is <empty> begin end ; 2) package One is package One -is <empty> begin end ; begin end ; 3) package One is <empty> package One -is <empty declaration> begin end ; begin end ; - - In parsing 3), the second 'package' causes a shift/reduce conflict; -shift to start the nested declaration (as in 2), reduce to the empty -declaration. Both are correct according to the grammar. - - Choice 2 leads to a shift/reduce conflict in the production for -package_body; implementing the wisi parser as a generalized LALR parser -allows it to handle this option. - - Choice 2 is the preferred choice for Ada, since it involves the least -modifications to the original Ada grammar in the Ada reference manual. - - -File: wisitoken-user_guide.info, Node: Grammar File Syntax, Prev: Common grammar problems, Up: Top - -3 Grammar File Syntax -********************* - -The grammar file syntax is based on Gnu bison syntax with some additions -and deletions (*note Bison: (bison)Top.). - - (The grammar is specified in the WisiToken grammar file -'wisitoken_grammar.wy'). - - The top level file structure is a list of declarations and -nonterminals. - - Comments are started by ';;' and terminated by end of line. - -* Menu: - -* Declarations:: -* Nonterminals:: -* Conditional code:: - - -File: wisitoken-user_guide.info, Node: Declarations, Next: Nonterminals, Up: Grammar File Syntax - -3.1 Declarations -================ - -Declarations declare terminal tokens, conflicts, and other parser -parameters. - -* Menu: - -* Raw Code:: -* Keywords:: -* Tokens:: -* Error recovery:: -* Other declarations:: - - -File: wisitoken-user_guide.info, Node: Raw Code, Next: Keywords, Up: Declarations - -3.1.1 Raw code --------------- - -%code { actions | copyright_license } [spec | body | context | pre | post]... %{ <output language code> }% - - Raw code declarations contain arbitrary code, copied verbatim into -the output. The keywords following '%code' determine where the section -is output. - - -File: wisitoken-user_guide.info, Node: Keywords, Next: Tokens, Prev: Raw Code, Up: Declarations - -3.1.2 Keywords --------------- - -%keyword <name> <string> - - example: -%keyword SEMICOLON ";" - - "Keywords" are reserved words or symbols in the target language; the -lexers recognize them by the given string. - - -File: wisitoken-user_guide.info, Node: Tokens, Next: Error recovery, Prev: Keywords, Up: Declarations - -3.1.3 Tokens ------------- - -%token < kind > name regexp - - example: -%token <symbol> IDENTIFIER %[ ... ]% -%token <punctuation> TICK "'" - - The syntax of the regular expression is determined by the lexer -generator. The meaning of 'kind' is determined by the lexer ('re2c' -ignores this), with the following defined by the WisiToken generator. -Other token kinds have no effect; they may be used for documentation. - -'<string-double>' - %token <string-double> STRING_LITERAL %[ ... ]% - A string of characters that have string syntax, with double quote - delimiters. - -'<string-single>' - %token <string-single> CHARACTER_LITERAL %[ ... ]% - A string of characters that have string syntax, with single quote - delimiters. - -'<new-line>' - %token <new-line> [\n] %[ ... ]% - Not used by the wisi lexer; required by the Ada lexer. The third - argument is the regular expression to recognize the entire comment. - -'<non-reporting>' - %token <non-reporting> WHITESPACE %[ [ \t] ]% - A token that is recognized by the lexer, but not returned to the - parser. - -'<delimited-text>' - %token <delimited-text> RAW_CODE "%{" "}%" - A token that contains arbitrary text, delimited by the two strings. - - -File: wisitoken-user_guide.info, Node: Error recovery, Next: Other declarations, Prev: Tokens, Up: Declarations - -3.1.4 Error recovery --------------------- - -The parser uses an error recovery algorithm when it encounters a syntax -error; if a solution is found, the parse continues. - - Error recovery uses multiple tasks to take advantage of multiple CPU -cores. Unfortunately, this means there is a race condition; the -solutions found can be delivered in different orders on different runs. -This matters because each solution results in a successful parse, -possibly with different actions (different indentation computed, for -example). Which solution finally succeeds depends on which are -terminated due to identical parser stacks, which in turn depends on the -order they were delivered. - - Once the syntax errors are fixed, only ambiguities in the grammar -itself can cause a similar problem. - - Several grammar file declarations set parameters for the error -recovery. If none of these parameters are present in the grammar file, -the generated parser does not do error recovery. - - The error recovery algorithm generates possible solutions based on -the parse state preceding the error point, by inserting, deleting, or -pushing back tokens. Each possible solution is given a cost, and -enqueued to be checked later. Solutions are checked in cost order -(lowest first). - -'%mckenzie_check_limit <limit>' - The number of tokens past the error point that must be parsed - successfully for a solution to be deemed successful. Smaller - values give faster recovery; larger values give better solutions. - Too large a value risks encountering another user error, making a - solution impossible. 3 or 4 works well in practice. - -'mckenzie_check_delta_limit <limit>' - When error recovery is entered with multiple parsers active, once a - solution has been found for one parser, the other parsers are - allowed to check only 'mckenzie_check_delta_limit' possible - solutions before they fail. This prevents long recovery times. - -'%mckenzie_cost_default <insert> <delete> <push back> <ignore check fail>' - McKenzie error recovery default costs for insert, delete, push back - single tokens, and for ignoring a semantic check failure; four - floating point numbers. - - "Push back" means undo parsing; remove tokens from the parse stack - and put them back into the input stream. This moves the - insert/delete point, allowing better solutions. The push back - default cost is also the undo reduce default cost. - - If not specified, costs are zero. Costs can be negative; they all - add linearly. - -'%mckenzie_cost_delete <token> <cost>' - McKenzie error recovery delete cost for a specific token. - -'%mckenzie_cost_fast_forward <cost>' - McKenzie error recovery cost for parsing ahead after fixing one - error, moving to the next error location. - -'%mckenzie_cost_insert <token> <cost>' - McKenzie error recovery insert cost for a specific token. - -'%mckenzie_cost_fast_forward <cost>' - McKenzie error recovery cost for using the 'matching_begin' - strategy. - -'%mckenzie_cost_push_back <token> <cost>' - McKenzie error recovery push back cost for a specific token. - -'%mckenzie_cost_undo_reduce <token> <cost>' - McKenzie error recovery undo reduce cost for a specific token. - -'%mckenzie_enqueue_limit <integer>' - McKenzie error recovery limit on possible solutions enqueued (to be - checked); default max integer. - -'%mckenzie_minimal_complete_cost_delta <cost>' - McKenzie error recovery cost added to the cost of an inserted token - when the insert is done by the minimal complete strategy; this cost - is normally negative. - - -File: wisitoken-user_guide.info, Node: Other declarations, Prev: Error recovery, Up: Declarations - -3.1.5 Other declarations ------------------------- - -'%case_insensitive' - If present, keywords are case insensitive in the lexer. - -'%conflict <conflict description>' - Declare a known conflict. - - Example conflict declaration: - %conflict REDUCE/REDUCE in state abstract_limited_opt, abstract_limited_synchronized_opt on token NEW - - The conflict description is output by 'wisitoken-bnf-generate' when - an undeclared conflict is detected. If the user decides to not fix - the conflict, the description can be copied into the grammar source - file, so it will be ignored next time around. - - Resolving conflicts in the grammar can be difficult, but leaving - them in can increase parse time and cause ambiguous parses. - -'%elisp_face <name>' - Declare a name for an elisp face constant. - - When generating Ada code for Emacs, the elisp faces applied by - 'wisi-face-apply' actions must be declared, so the elisp and Ada - code aggree on what they mean. - -'%elisp_indent <elisp name> <Ada name>' - Declare elisp and Ada names for an indent variable. - - When generating Ada code for Emacs, the elisp indent variables used - in 'wisi-indent' actions must be declared, so the elisp and Ada - code aggree on what they mean. - -'%elisp_action <elisp name> <Ada name>' - Declare elisp and Ada names for a custom action subprogram written - in Ada. - - The term "elisp" here is historical; the name is not actually used - by elisp in the current implementation. - -'end_names_optional_option <name>' - When generating Ada code for Emacs, the name of the Ada variable - determining whether end block names are optional. - - In the Ada language, block names can be repeated at the end; for - example: - - Get_Inputs : - loop - ... - end loop Get_Inputs; - - These names are optional in the Ada standard. Making them required - improves error recovery; the recovery algorithm can use matching - names to isolate the error. - -'generate <generate_algorithm> <output_language> [text_rep]' - - '<generate_algorithm>' is one of 'LALR | LR1 | Packrat_Gen | - Packrat_Proc | External' - - '<output_language>' is one of 'Ada | Ada_Emacs' - - The algorithm/output_language pair declares one output source set. - Multiple sets can be declared; they are all generated together. - - 'text_rep' determines how the parse table is represented; if - present, it is in a text file that is loaded at parser run time. - If absent, it is in the code. For very large parse tables, such as - for an LR1 parser for a large language like Ada, the text - representation may be needed, because the Ada compiler can't handle - the very large number of statements that represent the parser table - in the code. The text file can take a long time to read at parser - startup (a few seconds for the Ada language). - -'%language_runtime' - Specify an alternate name for the language runtime package; the - default is 'Wisi.<language_name>'. - -'%meta_syntax [BNF | EBNF]' - Declares the syntax used by the grammar file. BNF is a minor - extension of standard Backus Naur Form; EBNF is a large extension. - The default is BNF. - -'%no_enum' - By default, the generated Ada code includes an enumeration type - declaring each token. This makes the language-specific runtime - easier to write (without this type, tokens are identified by - integers). - - '%no_enum' causes the generated code to not include the token - enumeration type. - -'%no_language_runtime' - When generating Ada code for Emacs, '%no_language_runtime' causes - the generated code to not include the runtime. Some grammars may - need no runtime, particularly if they are small grammars intendend - to test some generator feature. - -'%partial_recursion' - The error recovery algorithm requires computing the recursion - present in the language grammar. For some grammars (such as Java), - this is too hard; '%partial_recursion' tells WisiToken to use a - simpler approximate calculation. This will affect the quality of - the error recovery, but it will still be robust. - -'%start' - The start token for the grammar. - -'re2c_regexp <name> <value>' - Declare a named regular expression with re2c name and syntax. The - name may then occur in another re2c regular expression. - - -File: wisitoken-user_guide.info, Node: Nonterminals, Next: Conditional code, Prev: Declarations, Up: Grammar File Syntax - -3.2 Nonterminals -================ - -A nonterminal statement declares a nonterminal token, and the associated -production rules and actions. - - The syntax of a nonterminal statement is: - -nonterminal : rhs {| rhs} ; - A nonterminal is defined by a list of alternate right hand sides. - -rhs : {rhs_item} [action [action]] ; - Each right hand side is a list of items, followed by zero to two -actions; the first is the post-parse action, the second the in-parse -action. - - In-parse actions are exeuted during the parse, when the production is -reduced; they can add semantic checks that help during error recovery. - - Post-parse actions are executed after the parse is complete, when a -node produced by this production is visited during the tree traversal; -they typically build an abstract syntax tree. - - The actions are written in output-language code; for 'Ada_Emacs' -output, this is elisp (a hold-over from when WisiToken only output elisp -code). - - If using BNF: -rhs_item : token ; - Where 'token' is defined by a token declaration. - - if using EBNF: -rhs_item - : token - | < identifier = identifier > - | rhs_optional_item - | rhs_multiple_item - | '(' rhs {| rhs} ')' - ; - Here 'token' is either defined by a token declaration, or the token -value contained in single quotes. - - The second option is an attribute, as defined by ANTLR; these are -ignored in wisitoken. - - Parentheses are used to group items. - -rhs_optional_item - : '[' rhs {| rhs} ']' - | '(' rhs {| rhs} ')' '?' - | token '?' - ; - These options all mean the same thing; the content is present zero or -one times. - -rhs_multiple_item - : '{' rhs {| rhs} '}' - | '{' rhs {| rhs} '}-' - | '(' rhs {| rhs} ')+' - | '(' rhs {| rhs} ')*' - | token '+' - | token '*' - ; - "{}", "()*", and "token*" mean the content is present zero or more -times. "{}-", "()+", and "token+" mean the content is present one or -more times. - - -File: wisitoken-user_guide.info, Node: Conditional code, Prev: Nonterminals, Up: Grammar File Syntax - -3.3 Conditional code -==================== - -It is sometimes necessary to include or exclude some declarations and -portions of rules based on the choice of lexer or parser. - - Therefore WisiToken supports '%if ... %end if' in the grammar file: -%if {lexer | parser} = {<lexer> | <generate_algorithm>} -... -%end if - - The lines between '%if' and '%end if' are ignored if the current -lexer or parser is not the one specified in the '%if' condition. - - '%if ... %end if' cannot be nested. - - - -Tag Table: -Node: Top727 -Node: Overview1378 -Node: Install2140 -Node: Common grammar problems2637 -Node: Empty choice in list2930 -Node: Grammar File Syntax5912 -Node: Declarations6469 -Node: Raw Code6775 -Node: Keywords7156 -Node: Tokens7468 -Node: Error recovery8803 -Node: Other declarations12525 -Node: Nonterminals17017 -Node: Conditional code19038 - -End Tag Table - - -Local Variables: -coding: utf-8 -End: diff --git a/wisitoken-user_guide.texinfo b/wisitoken-user_guide.texinfo new file mode 100644 index 0000000..0a371a8 --- /dev/null +++ b/wisitoken-user_guide.texinfo @@ -0,0 +1,616 @@ +\input texinfo +@c Author : Stephen Leake stephen_le...@stephe-leake.org +@c Web : http://stephe-leake.org/ada/opentoken.html +@setfilename wisitoken-user_guide.info +@settitle WisiToken User Guide + +@copying +Copyright @copyright{} 2014-2015, 2017, 2018, 2020, 2021 Stephen Leake. + +@quotation +Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, +Version 1.3 or any later version published by the Free Software +Foundation; with no Invariant Sections, no Front-Cover Texts and +no Back-Cover Texts. A copy of the license is included in the +section entitled "GNU Free Documentation License". +@end quotation +@end copying + +@dircategory Parser generators +@direntry +* wisitoken-bnf-generate: (wisitoken-bnf-generate). Ada parser generator +@end direntry + +@titlepage +@sp 10 +@title WisiToken User Guide +@page +@vskip 0pt plus 1filll +@insertcopying +@end titlepage + +@contents + +@node Top +@top WisiToken User Guide + +@ifnottex +@insertcopying +@end ifnottex + +@menu +* Overview:: +* Common grammar problems:: +* Grammar File Syntax:: +@end menu + +@node Overview +@chapter Overview + +WisiToken is a parser and parser generator toolkit, supporting +generalized LR (both LALR and LR1) and packrat parsers; the LR parser +provides robust error recovery. The grammar can be expressed as either +Ada source code statements, or in an EBNF file. The parser generator +generates Ada, either plain or assuming the Emacs wisi package. + +At one point, ``wisi'' was short for ``Wisent Indentation engine''; +the Emacs 'wisi' package implements an indentation engine that used to +be based on the Emacs wisent parser. However, that parser has now been +replaced by the WisiToken parser, so ``wisi'' is just a name. + +@menu +* Install:: +@end menu + +@node Install +@section Install +WisiToken is available as source code only, although a subset is +available in the GNU ELPA package @code{wisi}. + +You will also need to install a lexer generator. WisiToken supports +re2c, and other lexers can be added. + +re2c is available from @url{http://re2c.org/}; it is also packaged in +Mingw64 and Debian. WisiToken requires at least version 1.3. +WisiToken assumes the executable @code{re2c} is in +@code{$PATH}. + +@node Common grammar problems +@chapter Common grammar problems + +LALR grammars are tricky. Here we describe some common problems people +run into. + +@menu +* Empty choice in list:: +@end menu + +@node Empty choice in list +@section Empty choice in list + +Many programming languages have lists in the grammar. For example, Ada +has lists of declarations: + +@example +package_body + : PACKAGE name IS declaration_list BEGIN statement_list END SEMICOLON + ; + +declaration_list + : declaration + | declaration_list declaration + ; + +declaration + : object_declaration + | subprogram_declaration + ;; ... + ; +@end example + +Note that the above grammar fragment does not allow an empty +declaration_list. But Ada does, so the question is how can we add that +to the grammar. + +There are four choices: + +@enumerate +@item +Add an empty declaration choice to declaration_list: + +@example +declaration_list + : ;; empty list + | declaration + | declaration_list declaration + ; +@end example +This is now redundant; since declaration_list can be empty, the second +choice is not needed: +@example +declaration_list + : ;; empty list + | declaration_list declaration + ; +@end example + + +@item +Add an empty declaration choice to declaration: + +@example +declaration + : ;; empty declaration + | object_declaration + | subprogram_declaration + ;; ... + ; +@end example + +@item +Add another rule with the empty production: + +@example +package_body + : PACKAGE name IS declarative_part BEGIN statement_list END SEMICOLON + ; + +declarative_part + : ;; empty + | declaration_list + ; + +declaration_list + : declaration + | declaration_list declaration + ; + +declaration + : object_declaration + | subprogram_declaration + ;; ... + ; +@end example + +@item +Add another choice in package_body that leaves out declaration_list: +@example +package_body + : PACKAGE name IS declaration_list BEGIN statement_list END SEMICOLON + | PACKAGE name IS BEGIN statement_list END SEMICOLON + ; +@end example +@end enumerate + +Choice 1 is redundant, giving parse errors at parse time. +Consider the following statements, where "<empty>" is used to indicate +an empty declaration: + +1) package One is <empty> begin end ; +2) package One is package One is <empty> begin end ; begin end ; +3) package One is <empty> package One is <empty declaration> begin end ; begin end ; + +In parsing 3), the second 'package' causes a shift/reduce conflict; +shift to start the nested declaration (as in 2), reduce to the empty +declaration. Both are correct according to the grammar. + +Choice 2 leads to a shift/reduce conflict in the production for +package_body; implementing the wisi parser as a generalized LALR parser +allows it to handle this option. + +Choice 2 is the preferred choice for Ada, since it involves the least +modifications to the original Ada grammar in the Ada reference manual. + +@node Grammar File Syntax +@chapter Grammar File Syntax + +The grammar file syntax is based on Gnu bison syntax +with some additions and deletions (@pxref{Top,Bison,Overview,bison}). + +(The grammar is specified in the WisiToken grammar file +@file{wisitoken_grammar.wy}). + +The top level file structure is a list of declarations and +nonterminals. + +Comments are started by @code{;;} and terminated by end of line. + +@menu +* Declarations:: +* Nonterminals:: +* Conditional code:: +@end menu + +@node Declarations +@section Declarations + +Declarations declare terminal tokens, conflicts, and other parser +parameters. + +@menu +* Raw Code:: +* Keywords:: +* Tokens:: +* Error recovery:: +* Other declarations:: +@end menu + +@node Raw Code +@subsection Raw code +@verbatim +%code { actions | copyright_license } [spec | body | context | pre | post]... %{ <output language code> }% +@end verbatim + +Raw code declarations contain arbitrary code, copied verbatim into the +output. The keywords following @code{%code} determine where +the section is output. + +@node Keywords +@subsection Keywords +@verbatim +%keyword <name> <string> +@end verbatim + +example: +@verbatim +%keyword SEMICOLON ";" +@end verbatim + +``Keywords'' are reserved words or symbols in the target language; the +lexers recognize them by the given string. + +@node Tokens +@subsection Tokens +@verbatim +%token < kind > name regexp +@end verbatim + +example: +@verbatim +%token <symbol> IDENTIFIER %[ ... ]% +%token <punctuation> TICK "'" +@end verbatim + +The syntax of the regular expression is determined by the lexer +generator. The meaning of @code{kind} is determined by the lexer +(@code{re2c} ignores this), with the following defined by the +WisiToken generator. Other token kinds have no effect; they may be +used for documentation. + +@table @code +@item <string-double> +@verbatim +%token <string-double> STRING_LITERAL %[ ... ]% +@end verbatim +A string of characters that have string syntax, with double quote delimiters. + +@item <string-single> +@verbatim +%token <string-single> CHARACTER_LITERAL %[ ... ]% +@end verbatim +A string of characters that have string syntax, with single quote delimiters. + +@item <new-line> +@verbatim +%token <new-line> [\n] %[ ... ]% +@end verbatim +Not used by the wisi lexer; required by the Ada lexer. The third +argument is the regular expression to recognize the entire comment. + +@item <non-reporting> +@verbatim +%token <non-reporting> WHITESPACE %[ [ \t] ]% +@end verbatim +A token that is recognized by the lexer, but not returned to the +parser. + +@item <delimited-text> +@verbatim +%token <delimited-text> RAW_CODE "%{" "}%" +@end verbatim +A token that contains arbitrary text, delimited by the two strings. + +@end table + +@node Error recovery +@subsection Error recovery +The parser uses an error recovery algorithm when it encounters a +syntax error; if a solution is found, the parse continues. + +Error recovery uses multiple tasks to take advantage of multiple CPU +cores. Unfortunately, this means there is a race condition; the +solutions found can be delivered in different orders on different +runs. This matters because each solution results in a successful +parse, possibly with different actions (different indentation +computed, for example). Which solution finally succeeds depends on +which are terminated due to identical parser stacks, which in turn +depends on the order they were delivered. + +Once the syntax errors are fixed, only ambiguities in the grammar +itself can cause a similar problem. + +Several grammar file declarations set parameters for the error +recovery. If none of these parameters are present in the grammar file, +the generated parser does not do error recovery. + +The error recovery algorithm generates possible solutions based on the +parse state preceding the error point, by inserting, deleting, or pushing +back tokens. Each possible solution is given a cost, and enqueued to +be checked later. Solutions are checked in cost order (lowest first). + +@table @code +@item %mckenzie_check_limit <limit> +The number of tokens past the error point that must be parsed +successfully for a solution to be deemed successful. Smaller values +give faster recovery; larger values give better solutions. Too large a +value risks encountering another user error, making a solution +impossible. 3 or 4 works well in practice. + +@item mckenzie_check_delta_limit <limit> +When error recovery is entered with multiple parsers active, once a +solution has been found for one parser, the other parsers are allowed +to check only @code{mckenzie_check_delta_limit} possible solutions +before they fail. This prevents long recovery times. + +@item %mckenzie_cost_default <insert> <delete> <push back> <ignore check fail> +McKenzie error recovery default costs for insert, delete, push back +single tokens, and for ignoring a semantic check failure; four +floating point numbers. + +``Push back'' means undo parsing; remove tokens from the parse stack +and put them back into the input stream. This moves the insert/delete +point, allowing better solutions. The push back default cost is also +the undo reduce default cost. + +If not specified, costs are zero. Costs can be negative; they +all add linearly. + +@item %mckenzie_cost_delete <token> <cost> +McKenzie error recovery delete cost for a specific token. + +@item %mckenzie_cost_fast_forward <cost> +McKenzie error recovery cost for parsing ahead after fixing one error, +moving to the next error location. + +@item %mckenzie_cost_insert <token> <cost> +McKenzie error recovery insert cost for a specific token. + +@item %mckenzie_cost_fast_forward <cost> +McKenzie error recovery cost for using the @code{matching_begin} +strategy. + +@item %mckenzie_cost_push_back <token> <cost> +McKenzie error recovery push back cost for a specific token. + +@item %mckenzie_cost_undo_reduce <token> <cost> +McKenzie error recovery undo reduce cost for a specific token. + +@item %mckenzie_enqueue_limit <integer> +McKenzie error recovery limit on possible solutions enqueued (to be +checked); default max integer. + +@item %mckenzie_minimal_complete_cost_delta <cost> +McKenzie error recovery cost added to the cost of an inserted token +when the insert is done by the minimal complete strategy; this cost is +normally negative. + +@end table + +@node Other declarations +@subsection Other declarations +@table @code +@item %case_insensitive +If present, keywords are case insensitive in the lexer. + +@item %conflict <conflict description> +Declare a known conflict. + +Example conflict declaration: +@verbatim +%conflict REDUCE/REDUCE in state abstract_limited_opt, abstract_limited_synchronized_opt on token NEW +@end verbatim + +The conflict description is output by @code{wisitoken-bnf-generate} +when an undeclared conflict is detected. If the user decides to not +fix the conflict, the description can be copied into the grammar +source file, so it will be ignored next time around. + +Resolving conflicts in the grammar can be difficult, but leaving them +in can increase parse time and cause ambiguous parses. + +@item %elisp_face <name> +Declare a name for an elisp face constant. + +When generating Ada code for Emacs, the elisp faces applied by +@code{wisi-face-apply} actions must be declared, so the elisp and Ada +code aggree on what they mean. + +@item %elisp_indent <elisp name> <Ada name> +Declare elisp and Ada names for an indent variable. + +When generating Ada code for Emacs, the elisp indent variables used in +@code{wisi-indent} actions must be declared, so the elisp and Ada code +aggree on what they mean. + +@item %elisp_action <elisp name> <Ada name> +Declare elisp and Ada names for a custom action subprogram written in +Ada. + +The term ``elisp'' here is historical; the name is not actually used +by elisp in the current implementation. + +@item end_names_optional_option <name> +When generating Ada code for Emacs, the name of the Ada variable +determining whether end block names are optional. + +In the Ada language, block names can be repeated at the end; for +example: + +@verbatim +Get_Inputs : +loop +... +end loop Get_Inputs; +@end verbatim + +These names are optional in the Ada standard. Making them required +improves error recovery; the recovery algorithm can use matching names +to isolate the error. + +@item generate <generate_algorithm> <output_language> [text_rep] + +@code{<generate_algorithm>} is one of @code{LALR | LR1 | Packrat_Gen | Packrat_Proc | External} + +@code{<output_language>} is one of @code{Ada | Ada_Emacs} + +The algorithm/output_language pair declares one output source +set. Multiple sets can be declared; they are all generated together. + +@code{text_rep} determines how the parse table is represented; if +present, it is in a text file that is loaded at parser run time. If +absent, it is in the code. For very large parse tables, such as for an +LR1 parser for a large language like Ada, the text representation may +be needed, because the Ada compiler can't handle the very large number +of statements that represent the parser table in the code. The text +file can take a long time to read at parser startup (a few seconds for +the Ada language). + +@item %language_runtime +Specify an alternate name for the language runtime package; the +default is @code{Wisi.<language_name>}. + +@item %meta_syntax [BNF | EBNF] +Declares the syntax used by the grammar file. BNF is a minor extension +of standard Backus Naur Form; EBNF is a large extension. The default +is BNF. + +@item %no_enum +By default, the generated Ada code includes an enumeration type +declaring each token. This makes the language-specific runtime easier +to write (without this type, tokens are identified by integers). + +@code{%no_enum} causes the generated code to not include the token +enumeration type. + +@item %no_language_runtime +When generating Ada code for Emacs, @code{%no_language_runtime} causes +the generated code to not include the runtime. Some grammars may need +no runtime, particularly if they are small grammars intendend to test +some generator feature. + +@item %partial_recursion +The error recovery algorithm requires computing the recursion present +in the language grammar. For some grammars (such as Java), this is too +hard; @code{%partial_recursion} tells WisiToken to use a simpler +approximate calculation. This will affect the quality of the error +recovery, but it will still be robust. + +@item %start +The start token for the grammar. + +@item re2c_regexp <name> <value> +Declare a named regular expression with re2c name and syntax. The name +may then occur in another re2c regular expression. +@end table + +@node Nonterminals +@section Nonterminals + +A nonterminal statement declares a nonterminal token, and the +associated production rules and actions. + +The syntax of a nonterminal statement is: + +@verbatim +nonterminal : rhs {| rhs} ; +@end verbatim +A nonterminal is defined by a list of alternate right hand sides. + +@verbatim +rhs : {rhs_item} [action [action]] ; +@end verbatim +Each right hand side is a list of items, followed by zero to two +actions; the first is the post-parse action, the second the in-parse +action. + +In-parse actions are exeuted during the parse, when the production is +reduced; they can add semantic checks that help during error recovery. + +Post-parse actions are executed after the parse is complete, when a +node produced by this production is visited during the tree traversal; +they typically build an abstract syntax tree. + +The actions are written in output-language code; for @code{Ada_Emacs} +output, this is elisp (a hold-over from when WisiToken only output +elisp code). + +If using BNF: +@verbatim +rhs_item : token ; +@end verbatim +Where @code{token} is defined by a token declaration. + +if using EBNF: +@verbatim +rhs_item + : token + | < identifier = identifier > + | rhs_optional_item + | rhs_multiple_item + | '(' rhs {| rhs} ')' + ; +@end verbatim +Here @code{token} is either defined by a token +declaration, or the token value contained in single quotes. + +The second option is an attribute, as defined by ANTLR; these are +ignored in wisitoken. + +Parentheses are used to group items. + +@verbatim +rhs_optional_item + : '[' rhs {| rhs} ']' + | '(' rhs {| rhs} ')' '?' + | token '?' + ; +@end verbatim +These options all mean the same thing; the content is present zero or +one times. + +@verbatim +rhs_multiple_item + : '{' rhs {| rhs} '}' + | '{' rhs {| rhs} '}-' + | '(' rhs {| rhs} ')+' + | '(' rhs {| rhs} ')*' + | token '+' + | token '*' + ; +@end verbatim +''@{@}'', ''()*'', and ''token*'' mean the content is present zero or more +times. ''@{@}-'', ''()+'', and ''token+'' mean the content is present one or +more times. + +@node Conditional code +@section Conditional code + +It is sometimes necessary to include or exclude some declarations +and portions of rules based on the choice of lexer or parser. + +Therefore WisiToken supports @code{%if ... %end if} in the grammar file: +@verbatim +%if {lexer | parser} = {<lexer> | <generate_algorithm>} +... +%end if +@end verbatim + +The lines between @code{%if} and @code{%end if} are ignored if the +current lexer or parser is not the one specified in the @code{%if} +condition. + +@code{%if ... %end if} cannot be nested. + +@c FIXME: doc language_fixes etc. +@bye